Smalltalk › Squeak › Squeak VM

FFI Struct Argument Pass By Value Fails on Mac 64 bit

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

13 messages Options

tblanchard

FFI Struct Argument Pass By Value Fails on Mac 64 bit

I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

I'm stuck. The FFI64 plugin is way beyond my comprehension.

// microclang.c

typedef struct {

  const void *ptr_data[2];

  unsigned int_data;

} CXSourceLocation;

/**

 * \brief Identifies a half-open character range in the source code.

*

 * Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the

 * starting and end locations from a source range, respectively.

*/

typedef struct {

  const void *ptr_data[2];

  unsigned begin_int_data;

  unsigned end_int_data;

} CXSourceRange;

const char* first = "first_pointer";

const char* second = "second_pointer";

// return a fake range with non zero data

CXSourceRange clang_getArbitraryRange()

{

  CXSourceRange range = {0};

  range.ptr_data[0] = (void*)first;

  range.ptr_data[1] = (void*)second;

  range.begin_int_data = 17;

  range.end_int_data = 24;

  return range;

}

// Actual clang function - range is always zero'd here despite having values in the image

CXSourceLocation clang_getRangeStart(CXSourceRange range) {

  // Special decoding for CXSourceLocations for CXLoadedDiagnostics.

  if ((uintptr_t)range.ptr_data[0] & 0x1) {

    CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 };

    return Result;    

}

  CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] },

    range.begin_int_data };

  return Result;

}

LibCClang-FFI-Binding.st (7K) Download Attachment

microclang.c (1K) Download Attachment

Ben Coman

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

Hi Todd,

Could you advise the the Playground code to use to test this?

And maybe a C program that links to the microclang shared library that does the equivalent test?

cheers -ben

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:

I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

I'm stuck. The FFI64 plugin is way beyond my comprehension.

// microclang.c

typedef struct {
const void *ptr_data[2];
unsigned int_data;
} CXSourceLocation;

/**
* \brief Identifies a half-open character range in the source code.
*
* Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the
* starting and end locations from a source range, respectively.
*/
typedef struct {
const void *ptr_data[2];
unsigned begin_int_data;
unsigned end_int_data;
} CXSourceRange;

const char* first = "first_pointer";
const char* second = "second_pointer";

// return a fake range with non zero data
CXSourceRange clang_getArbitraryRange()
{
CXSourceRange range = {0};
range.ptr_data[0] = (void*)first;
range.ptr_data[1] = (void*)second;
range.begin_int_data = 17;
range.end_int_data = 24;
return range;
}

// Actual clang function - range is always zero'd here despite having values in the image
CXSourceLocation clang_getRangeStart(CXSourceRange range) {
// Special decoding for CXSourceLocations for CXLoadedDiagnostics.
if ((uintptr_t)range.ptr_data[0] & 0x1) {
CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 };
return Result;
}

CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] },
range.begin_int_data };
return Result;
}

tblanchard

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

Playground code:

LibCClang startFromRange:(LibCClang defaultRange)

it worked if the resulting struct is not all zeroes.

A tiny C program is the equivalent.

int main()

{

if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0) printf("That failed\n");

else printf("That worked\n")

return 0;

}

On Nov 21, 2017, at 11:18 PM, Ben Coman <[hidden email]> wrote:

Hi Todd,

Could you advise the the Playground code to use to test this?
And maybe a C program that links to the microclang shared library that does the equivalent test?

cheers -ben

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:

I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

I'm stuck. The FFI64 plugin is way beyond my comprehension.

// microclang.c

typedef struct {
const void *ptr_data[2];
unsigned int_data;
} CXSourceLocation;

/**
* \brief Identifies a half-open character range in the source code.
*
* Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the
* starting and end locations from a source range, respectively.
*/
typedef struct {
const void *ptr_data[2];
unsigned begin_int_data;
unsigned end_int_data;
} CXSourceRange;

const char* first = "first_pointer";
const char* second = "second_pointer";

// return a fake range with non zero data
CXSourceRange clang_getArbitraryRange()
{
CXSourceRange range = {0};
range.ptr_data[0] = (void*)first;
range.ptr_data[1] = (void*)second;
range.begin_int_data = 17;
range.end_int_data = 24;
return range;
}

// Actual clang function - range is always zero'd here despite having values in the image
CXSourceLocation clang_getRangeStart(CXSourceRange range) {
// Special decoding for CXSourceLocations for CXLoadedDiagnostics.
if ((uintptr_t)range.ptr_data[0] & 0x1) {
CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 };
return Result;
}

CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] },
range.begin_int_data };
return Result;
}

Ben Coman

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

In reply to this post by tblanchard

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:

I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...

$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang

test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'

if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)

I presume you meant...

if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0)

so correcting and continuing...

$ clang test.c -L. -l microclang

$ LD_LIBRARY_PATH=. ./a.out

That failed

So I'm not sure how to proceed.

I was expecting that would work while Pharo failed.

Now interestingly...

$ clang test.c microlang.c

$ ./a.out

That worked

So it seems a similar problem exists outside our FFI.

cheers -ben

P.S. I refactored you code to extract a header file (attached)

I'm stuck. The FFI64 plugin is way beyond my comprehension.

// microclang.c

typedef struct {
const void *ptr_data[2];
unsigned int_data;
} CXSourceLocation;

/**
* \brief Identifies a half-open character range in the source code.
*
* Use clang_getRangeStart() and clang_getRangeEnd() to retrieve the
* starting and end locations from a source range, respectively.
*/
typedef struct {
const void *ptr_data[2];
unsigned begin_int_data;
unsigned end_int_data;
} CXSourceRange;

const char* first = "first_pointer";
const char* second = "second_pointer";

// return a fake range with non zero data
CXSourceRange clang_getArbitraryRange()
{
CXSourceRange range = {0};
range.ptr_data[0] = (void*)first;
range.ptr_data[1] = (void*)second;
range.begin_int_data = 17;
range.end_int_data = 24;
return range;
}

// Actual clang function - range is always zero'd here despite having values in the image
CXSourceLocation clang_getRangeStart(CXSourceRange range) {
// Special decoding for CXSourceLocations for CXLoadedDiagnostics.
if ((uintptr_t)range.ptr_data[0] & 0x1) {
CXSourceLocation Result = { { range.ptr_data[0], nullptr }, 0 };
return Result;
}

CXSourceLocation Result = { { range.ptr_data[0], range.ptr_data[1] },
range.begin_int_data };
return Result;
}

clang_ffi_struct_test.zip (2K) Download Attachment

Ben Coman

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:

I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)

I presume you meant...
if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0)
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked

So it seems a similar problem exists outside our FFI.

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.

Attached zip exploded here for easy reference...

___microlang.h___

typedef unsigned uintptr_t;

typedef struct {

const void *ptr_data[2];

} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();

int clang_getRangeEnd_(CXSourceRange_ range);

___microclang.c___

#include "microclang.h"

const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()

{ CXSourceRange_ range = {0};

range.ptr_data[0] = (void*)libraryString;

return range;

}

int clang_getRangeEnd_(CXSourceRange_ range)

{ // Special decoding for CXSourceLocations for CXLoadedDiagnostics.

if ((uintptr_t)range.ptr_data[0] & 0x1)

{ return 0; }

else

{ return 1; }

}

___test.c___

#include <stdio.h>

#include "microclang.h"

const char* localString = "local_pointer";

void test( CXSourceRange_ range, char *note )

{ int result = clang_getRangeEnd_(range);

if(result == 0)

{ printf("That failed (%s)\n", note); }

else

{ printf("That worked (%s)\n", note); }

}

int main()

{ CXSourceRange_ range1 = clang_getArbitraryRange_();

test(range1, "library string");

CXSourceRange_ range2 = {0};

range2.ptr_data[0] = (void*)localString;

test(range2, "local string");

}

___Makefile___

default: clean static shared

clean:

rm -f *so *App

@echo

shared:

clang -g -o libmicroclang.so -shared -fPIC microclang.c

clang -g -o sharedApp test.c -L. -lmicroclang

LD_LIBRARY_PATH=. ./sharedApp

@echo

static:

clang -g -o staticApp test.c microclang.c

./staticApp

@echo

Now running...

$ make > report

gives...

___report___

rm -f *so *App

clang -g -o staticApp test.c microclang.c

./staticApp

That worked (library string)

That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c

clang -g -o sharedApp test.c -L. -lmicroclang

LD_LIBRARY_PATH=. ./sharedApp

That failed (library string)

That worked (local string)

cheers -ben

clang2.zip (2K) Download Attachment

Ben Coman

Re: FFI Struct Argument Pass By Value Fails on Mac 64 bit

On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:

I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)

I presume you meant...
if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0)
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked

So it seems a similar problem exists outside our FFI.

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...

___microlang.h___
typedef unsigned uintptr_t;

typedef struct {
const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);

___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{ CXSourceRange_ range = {0};
range.ptr_data[0] = (void*)libraryString;
return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{ // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
if ((uintptr_t)range.ptr_data[0] & 0x1)
{ return 0; }
else
{ return 1; }
}

___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString = "local_pointer";

void test( CXSourceRange_ range, char *note )
{ int result = clang_getRangeEnd_(range);
if(result == 0)
{ printf("That failed (%s)\n", note); }
else
{ printf("That worked (%s)\n", note); }
}

int main()
{ CXSourceRange_ range1 = clang_getArbitraryRange_();
test(range1, "library string");

CXSourceRange_ range2 = {0};
range2.ptr_data[0] = (void*)localString;
test(range2, "local string");
}

___Makefile___
default: clean static shared

clean:
rm -f *so *App
@echo

shared:
clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
@echo

static:
clang -g -o staticApp test.c microclang.c
./staticApp
@echo

Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)

Further simplification dealing *only* with strings (see attached sharedLibString.zip)

(also attached is clang3.zip as a step along the way)

___microclang.c___

typedef unsigned uintptr_t;

const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString()

{ return myLibraryString;

}

int lib_testString( const char *aString )

{ unsigned test = (uintptr_t)aString & 0x1;

printf("\n test=%d, aString-->%d\n", test, (uintptr_t)aString);

if (test)

{ return 0; }

else

{ return 1; }

}

___test.c___

#include <stdio.h>

int lib_testString( const char *aString );

const char *lib_getLibraryString();

const char *localString = "local_pointer";

void test( int result, char *note )

{ if(result == 0)

{ printf("That failed (%s)\n", note); }

else

{ printf("That worked (%s)\n", note); }

}

int main()

{ const char * libraryString = lib_getLibraryString();

test(lib_testString(libraryString), "library string");

test(lib_testString(localString), "local string");

}

$ make > report

___report___

rm -f *so *App

clang -g -o staticApp test.c microclang.c

./staticApp

test=0, aString-->4196150

That worked (library string)

test=0, aString-->4196068

That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c

clang -g -o sharedApp test.c -L. -lmicroclang

LD_LIBRARY_PATH=. ./sharedApp

test=1, aString-->-792512599

That failed (library string)

test=0, aString-->4196484

That worked (local string)

cheers -ben

sharedLibString.zip (2K) Download Attachment

clang3.zip (2K) Download Attachment

tblanchard

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

i'm getting the idea that we should probably write a test suite/library for FFI

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:

On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:

I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)

I presume you meant...
  if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0)
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked

So it seems a similar problem exists outside our FFI.

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...

___microlang.h___
typedef unsigned uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);

___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{ CXSourceRange_ range = {0};
  range.ptr_data[0] = (void*)libraryString;
  return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{ // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
  if ((uintptr_t)range.ptr_data[0] & 0x1)
  { return 0; }
  else
  { return 1; }
}

___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString = "local_pointer";

void test( CXSourceRange_ range, char *note )
{ int result = clang_getRangeEnd_(range);
  if(result == 0)
  { printf("That failed (%s)\n", note); }
  else
  { printf("That worked (%s)\n", note); }
}

int main()
{ CXSourceRange_ range1 = clang_getArbitraryRange_();
  test(range1, "library string");

  CXSourceRange_ range2 = {0};
  range2.ptr_data[0] = (void*)localString;
  test(range2, "local string");
}

___Makefile___
default: clean static shared

clean:
  rm -f *so *App
  @echo

shared:
  clang -g -o libmicroclang.so -shared -fPIC microclang.c
  clang -g -o sharedApp test.c -L. -lmicroclang
  LD_LIBRARY_PATH=. ./sharedApp
  @echo

static:
  clang -g -o staticApp test.c microclang.c
  ./staticApp
  @echo

Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)

Further simplification dealing *only* with strings (see attached sharedLibString.zip)
(also attached is clang3.zip as a step along the way)

___microclang.c___
typedef unsigned uintptr_t;
const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString()
{ return myLibraryString;
}

int lib_testString( const char *aString )
{ unsigned test = (uintptr_t)aString & 0x1;
printf("\n test=%d, aString-->%d\n", test, (uintptr_t)aString);
if (test)
{ return 0; }
else
{ return 1; }
}

___test.c___
#include <stdio.h>
int lib_testString( const char *aString );
const char *lib_getLibraryString();
const char *localString = "local_pointer";

void test( int result, char *note )
{ if(result == 0)
{ printf("That failed (%s)\n", note); }
else
{ printf("That worked (%s)\n", note); }
}

int main()
{ const char * libraryString = lib_getLibraryString();
test(lib_testString(libraryString), "library string");

test(lib_testString(localString), "local string");
}

$ make > report

___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp

test=0, aString-->4196150
That worked (library string)

test=0, aString-->4196068
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp

test=1, aString-->-792512599
That failed (library string)

test=0, aString-->4196484
That worked (local string)

cheers -ben

<sharedLibString.zip><clang3.zip>

Ben Coman

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

On 27 November 2017 at 00:24, Todd Blanchard <[hidden email]> wrote:

i'm getting the idea that we should probably write a test suite/library for FFI

I noticed these...

* https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/plugins/SqueakFFIPrims/ffi-test-main.c

* https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/Cross/plugins/SqueakFFIPrims/sqFFITestFuncs.c

* https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/Cross/plugins/IA32ABI/AlienSUnitTestProcedures.c

These look like c-code test frames for FFI. Could these be built by the OpenSmalltalk CI to be normally shipped with the VM so that Image-side CI can test against them?

And also some general info...

* https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/plugins/SqueakFFIPrims/00README

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:

On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:

I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

My last analysis discovered something interesting about strings defined inside shared libraries being handled differently, but only later realised I had chased the wrong rabbit down the hole.

I've now investigated the premise you actually poses, and I agree, that structs being zero is some cases.

The attached zipfile containing libstruct.c a few comparison cases - three "good" layouts that work fine and one "bad" that mostly gets zeros but sometimes other weird numbers. There is a Makefile with three main targets:

1. make layout - statically compiles libstruct.c and runs produced a.out to display structure layouts

2. make run - downloads Pharo, starts it loading LibStruct.st , then you manually run LibStruct>>>LibStructTest>>#testStructs and observe structure values on console

3. make debug - starts LLDB to run Pharo with breakpoints pre-configured for when you run #testStructs.

Note Pharo will freeze and you need to move to LLDB. Try these commands...

frame variable

call print_struct(&GoodStruct1_fmt, &aStruct)

continue

That works on Ubuntu 16.04 64 bit.

You will need to tune it for OSX.

clang and lldb are required.

================================

The offsets configured in the class variables of all ExternalStructs

correctly matches that reported by the C code test frame results here...

$ make layout

clang -g libstruct.c

./a.out

GoodStruct1:

uint32_t:int1: 01 4

uint32_t:int2: 05 4

GoodStruct2:

uint32_t:int1: 01 4

uint32_t:int2: 05 4

void*:ptr_data: 09 8

GoodStruct3:

void*:ptr_data: 01 8

uint32_t:int1: 09 4

uint32_t:int2: 13 4

BadStruct:

void*:ptr_data1: 01 8

void*:ptr_data2: 09 8

uint32_t:int1: 17 4

uint32_t:int2: 21 4

================================

$ make run

clang -g -o libstruct.so -shared -fPIC libstruct.c

getpharo/pharo-vm/lib/pharo/5.0-201707201942/pharo getpharo/Pharo.image ../LibStruct.st

Now manually browse to and run LibStruct>>>LibStructTest>>#testStructs

LibStructTest>>testStructs

|good1Struct good2Struct good3Struct badStruct |

good1Struct := GoodStruct1 new int1: 2; int2: 3.

good2Struct := GoodStruct2 new int1: 2; int2: 3; ptr_force_int: 4.

good3Struct := GoodStruct3 new int1: 2; int2: 3; ptr_force_int: 4.

badStruct := BadStruct new int1: 2; int2: 3; ptr_force_int1: 4; ptr_force_int1: 5.

self assert: (LibStruct tryGood1: good1Struct) equals: 6.

self assert: (LibStruct tryGood2: good2Struct) equals: 6.

self assert: (LibStruct tryGood3: good3Struct) equals: 6.

"Problem exposed in next line"

self assert: (LibStruct tryBad: badStruct) equals: 6.

which on console produces...

GoodStruct1:

uint32_t:int1: 01 4 = 02 00 00 00

uint32_t:int2: 05 4 = 03 00 00 00

GoodStruct2:

uint32_t:int1: 01 4 = 02 00 00 00

uint32_t:int2: 05 4 = 03 00 00 00

void*:ptr_data: 09 8 = 04 00 00 00 00 00 00 00

GoodStruct3:

void*:ptr_data: 01 8 = 04 00 00 00 00 00 00 00

uint32_t:int1: 09 4 = 02 00 00 00

uint32_t:int2: 13 4 = 03 00 00 00

BadStruct:

void*:ptr_data1: 01 8 = 00 00 00 00 00 00 00 00

void*:ptr_data2: 09 8 = 00 00 00 00 00 00 00 00

uint32_t:int1: 17 4 = 29 03 03 00

uint32_t:int2: 21 4 = 00 00 00 00

Comparing GoodStruct3 and BadStruct, it seems one pointer is handled fine, but not two.

The first time this is run after Image boots seems like BadStruct gets some random data.

================================

In same image, subsequent runs of LibStruct>>>LibStructTest>>testStructs

give only all zeros for BadStruct, as Todd observed.

GoodStruct1:

uint32_t:int1: 01 4 = 02 00 00 00

uint32_t:int2: 05 4 = 03 00 00 00

GoodStruct2:

uint32_t:int1: 01 4 = 02 00 00 00

uint32_t:int2: 05 4 = 03 00 00 00

void*:ptr_data: 09 8 = 00 00 00 00 00 00 00 00

GoodStruct3:

void*:ptr_data: 01 8 = 00 00 00 00 00 00 00 00

uint32_t:int1: 09 4 = 02 00 00 00

uint32_t:int2: 13 4 = 03 00 00 00

BadStruct:

void*:ptr_data1: 01 8 = 00 00 00 00 00 00 00 00

void*:ptr_data2: 09 8 = 00 00 00 00 00 00 00 00

uint32_t:int1: 17 4 = 00 00 00 00

uint32_t:int2: 21 4 = 00 00 00 00

cheers -ben

FFI64StructTest.zip (4K) Download Attachment

tblanchard

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

I got pulled off to do some $ work and haven't got back to this but I would like to get back to it.

Has there been any progress on this in the publicly available 64bit VM?

I see there is a completely separate FFI plugin for 32 vs 64 and worry 64 is still not quite ready for real work.

On Dec 3, 2017, at 7:18 AM, Ben Coman <[hidden email]> wrote:

On 27 November 2017 at 00:24, Todd Blanchard <[hidden email]> wrote:

i'm getting the idea that we should probably write a test suite/library for FFI

I noticed these...
* https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/plugins/SqueakFFIPrims/ffi-test-main.c
* https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/Cross/plugins/SqueakFFIPrims/sqFFITestFuncs.c
* https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/Cross/plugins/IA32ABI/AlienSUnitTestProcedures.c

These look like c-code test frames for FFI. Could these be built by the OpenSmalltalk CI to be normally shipped with the VM so that Image-side CI can test against them?

And also some general info...
* https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/plugins/SqueakFFIPrims/00README

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:

On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:

I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

My last analysis discovered something interesting about strings defined inside shared libraries being handled differently, but only later realised I had chased the wrong rabbit down the hole.

I've now investigated the premise you actually poses, and I agree, that structs being zero is some cases.
The attached zipfile containing libstruct.c a few comparison cases - three "good" layouts that work fine and one "bad" that mostly gets zeros but sometimes other weird numbers. There is a Makefile with three main targets:
1. make layout - statically compiles libstruct.c and runs produced a.out to display structure layouts

2. make run - downloads Pharo, starts it loading  LibStruct.st , then you manually run LibStruct>>>LibStructTest>>#testStructs and observe structure values on console

3. make debug - starts LLDB to run Pharo with breakpoints pre-configured for when you run #testStructs.
Note Pharo will freeze and you need to move to LLDB. Try these commands...
frame variable
call print_struct(&GoodStruct1_fmt, &aStruct)
continue

That works on Ubuntu 16.04 64 bit.
You will need to tune it for OSX.
clang and lldb are required.

================================
The offsets configured in the class variables of all ExternalStructs
correctly matches that reported by the C code test frame results here...

$ make layout
clang -g libstruct.c
./a.out

GoodStruct1:
uint32_t:int1: 01 4
uint32_t:int2: 05 4
GoodStruct2:
uint32_t:int1: 01 4
uint32_t:int2: 05 4
void*:ptr_data: 09 8
GoodStruct3:
void*:ptr_data: 01 8
uint32_t:int1: 09 4
uint32_t:int2: 13 4
BadStruct:
void*:ptr_data1: 01 8
void*:ptr_data2: 09 8
uint32_t:int1: 17 4
uint32_t:int2: 21 4

================================

$ make run
clang -g -o libstruct.so -shared -fPIC libstruct.c
getpharo/pharo-vm/lib/pharo/5.0-201707201942/pharo getpharo/Pharo.image ../LibStruct.st

Now manually browse to and run LibStruct>>>LibStructTest>>#testStructs
LibStructTest>>testStructs
|good1Struct good2Struct good3Struct badStruct |

good1Struct := GoodStruct1 new int1: 2; int2: 3.
good2Struct := GoodStruct2 new int1: 2; int2: 3; ptr_force_int: 4.
good3Struct := GoodStruct3 new int1: 2; int2: 3; ptr_force_int: 4.
badStruct := BadStruct new int1: 2; int2: 3; ptr_force_int1: 4; ptr_force_int1: 5.

self assert: (LibStruct tryGood1: good1Struct) equals: 6.
self assert: (LibStruct tryGood2: good2Struct) equals: 6.
self assert: (LibStruct tryGood3: good3Struct) equals: 6.
"Problem exposed in next line"
self assert: (LibStruct tryBad: badStruct) equals: 6.

which on console produces...

GoodStruct1:
uint32_t:int1: 01 4 = 02 00 00 00
uint32_t:int2: 05 4 = 03 00 00 00
GoodStruct2:
uint32_t:int1: 01 4 = 02 00 00 00
uint32_t:int2: 05 4 = 03 00 00 00
void*:ptr_data: 09 8 = 04 00 00 00 00 00 00 00
GoodStruct3:
void*:ptr_data: 01 8 = 04 00 00 00 00 00 00 00
uint32_t:int1: 09 4 = 02 00 00 00
uint32_t:int2: 13 4 = 03 00 00 00
BadStruct:
void*:ptr_data1: 01 8 = 00 00 00 00 00 00 00 00
void*:ptr_data2: 09 8 = 00 00 00 00 00 00 00 00
uint32_t:int1: 17 4 = 29 03 03 00
uint32_t:int2: 21 4 = 00 00 00 00

Comparing GoodStruct3 and BadStruct, it seems one pointer is handled fine, but not two.
The first time this is run after Image boots seems like BadStruct gets some random data.

================================
In same image, subsequent runs of LibStruct>>>LibStructTest>>testStructs
give only all zeros for BadStruct, as Todd observed.

GoodStruct1:
uint32_t:int1: 01 4 = 02 00 00 00
uint32_t:int2: 05 4 = 03 00 00 00
GoodStruct2:
uint32_t:int1: 01 4 = 02 00 00 00
uint32_t:int2: 05 4 = 03 00 00 00
void*:ptr_data: 09 8 = 00 00 00 00 00 00 00 00
GoodStruct3:
void*:ptr_data: 01 8 = 00 00 00 00 00 00 00 00
uint32_t:int1: 09 4 = 02 00 00 00
uint32_t:int2: 13 4 = 03 00 00 00
BadStruct:
void*:ptr_data1: 01 8 = 00 00 00 00 00 00 00 00
void*:ptr_data2: 09 8 = 00 00 00 00 00 00 00 00
uint32_t:int1: 17 4 = 00 00 00 00
uint32_t:int2: 21 4 = 00 00 00 00

cheers -ben

<FFI64StructTest.zip>

Eliot Miranda-2

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

In reply to this post by tblanchard

Hi Todd,

On Sun, Nov 26, 2017 at 8:24 AM, Todd Blanchard <[hidden email]> wrote:

i'm getting the idea that we should probably write a test suite/library for FFI

See the package FFI-Tests at http://source.squeak.org/FFI and the file sqFFITestFuncs.c

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:

On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:

I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)

I presume you meant...
  if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0)
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked

So it seems a similar problem exists outside our FFI.

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...

___microlang.h___
typedef unsigned uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);

___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{ CXSourceRange_ range = {0};
  range.ptr_data[0] = (void*)libraryString;
  return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{ // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
  if ((uintptr_t)range.ptr_data[0] & 0x1)
  { return 0; }
  else
  { return 1; }
}

___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString = "local_pointer";

void test( CXSourceRange_ range, char *note )
{ int result = clang_getRangeEnd_(range);
  if(result == 0)
  { printf("That failed (%s)\n", note); }
  else
  { printf("That worked (%s)\n", note); }
}

int main()
{ CXSourceRange_ range1 = clang_getArbitraryRange_();
  test(range1, "library string");

  CXSourceRange_ range2 = {0};
  range2.ptr_data[0] = (void*)localString;
  test(range2, "local string");
}

___Makefile___
default: clean static shared

clean:
  rm -f *so *App
  @echo

shared:
  clang -g -o libmicroclang.so -shared -fPIC microclang.c
  clang -g -o sharedApp test.c -L. -lmicroclang
  LD_LIBRARY_PATH=. ./sharedApp
  @echo

static:
  clang -g -o staticApp test.c microclang.c
  ./staticApp
  @echo

Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)

Further simplification dealing *only* with strings (see attached sharedLibString.zip)
(also attached is clang3.zip as a step along the way)

___microclang.c___
typedef unsigned uintptr_t;
const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString()
{ return myLibraryString;
}

int lib_testString( const char *aString )
{ unsigned test = (uintptr_t)aString & 0x1;
printf("\n test=%d, aString-->%d\n", test, (uintptr_t)aString);
if (test)
{ return 0; }
else
{ return 1; }
}

___test.c___
#include <stdio.h>
int lib_testString( const char *aString );
const char *lib_getLibraryString();
const char *localString = "local_pointer";

void test( int result, char *note )
{ if(result == 0)
{ printf("That failed (%s)\n", note); }
else
{ printf("That worked (%s)\n", note); }
}

int main()
{ const char * libraryString = lib_getLibraryString();
test(lib_testString(libraryString), "library string");

test(lib_testString(localString), "local string");
}

$ make > report

___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp

test=0, aString-->4196150
That worked (library string)

test=0, aString-->4196068
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp

test=1, aString-->-792512599
That failed (library string)

test=0, aString-->4196484
That worked (local string)

cheers -ben

<sharedLibString.zip><clang3.zip>

_,,,^..^,,,_

best, Eliot

Eliot Miranda-2

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

Todd,

On Thu, Mar 1, 2018 at 10:01 AM, Eliot Miranda <[hidden email]> wrote:

Hi Todd,

On Sun, Nov 26, 2017 at 8:24 AM, Todd Blanchard <[hidden email]> wrote:

i'm getting the idea that we should probably write a test suite/library for FFI

See the package FFI-Tests at http://source.squeak.org/FFI and the file sqFFITestFuncs.c

I suggest you extend these with an example that demonstrates your failing case. One thing to be very careful about is the platform. Are you on Mac OS X, linux or Win64? Mac OS X & linux use the SysV ABI (see e.g. https://software.intel.com/sites/default/files/article/402129/mpx-linux64-abi.pdf) whereas Win64 uses Microsoft's own ABI (see https://msdn.microsoft.com/en-us/library/ms235286.aspx?f=255&MSPPError=-2147217396). In particular under the Sys V ABI certain structs (structs having two 64-bit fields) will be passed in registers, if two registers are available at that point in the parameter list. The ThreadedX64SysVFFIPlugin is written to pass structs in this manner but there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion. Microsoft's own ABI passes structs of size 8, 16, 32, or 64 bits and __m64 in a single register. The ThreadedX64Win64FFIPlugin is written to pass structs in this manner but again there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion.

HTH

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:

On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:

I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)

I presume you meant...
  if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0)
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked

So it seems a similar problem exists outside our FFI.

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...

___microlang.h___
typedef unsigned uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);

___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{ CXSourceRange_ range = {0};
  range.ptr_data[0] = (void*)libraryString;
  return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{ // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
  if ((uintptr_t)range.ptr_data[0] & 0x1)
  { return 0; }
  else
  { return 1; }
}

___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString = "local_pointer";

void test( CXSourceRange_ range, char *note )
{ int result = clang_getRangeEnd_(range);
  if(result == 0)
  { printf("That failed (%s)\n", note); }
  else
  { printf("That worked (%s)\n", note); }
}

int main()
{ CXSourceRange_ range1 = clang_getArbitraryRange_();
  test(range1, "library string");

  CXSourceRange_ range2 = {0};
  range2.ptr_data[0] = (void*)localString;
  test(range2, "local string");
}

___Makefile___
default: clean static shared

clean:
  rm -f *so *App
  @echo

shared:
  clang -g -o libmicroclang.so -shared -fPIC microclang.c
  clang -g -o sharedApp test.c -L. -lmicroclang
  LD_LIBRARY_PATH=. ./sharedApp
  @echo

static:
  clang -g -o staticApp test.c microclang.c
  ./staticApp
  @echo

Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)

Further simplification dealing *only* with strings (see attached sharedLibString.zip)
(also attached is clang3.zip as a step along the way)

___microclang.c___
typedef unsigned uintptr_t;
const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString()
{ return myLibraryString;
}

int lib_testString( const char *aString )
{ unsigned test = (uintptr_t)aString & 0x1;
printf("\n test=%d, aString-->%d\n", test, (uintptr_t)aString);
if (test)
{ return 0; }
else
{ return 1; }
}

___test.c___
#include <stdio.h>
int lib_testString( const char *aString );
const char *lib_getLibraryString();
const char *localString = "local_pointer";

void test( int result, char *note )
{ if(result == 0)
{ printf("That failed (%s)\n", note); }
else
{ printf("That worked (%s)\n", note); }
}

int main()
{ const char * libraryString = lib_getLibraryString();
test(lib_testString(libraryString), "library string");

test(lib_testString(localString), "local string");
}

$ make > report

___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp

test=0, aString-->4196150
That worked (library string)

test=0, aString-->4196068
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp

test=1, aString-->-792512599
That failed (library string)

test=0, aString-->4196484
That worked (local string)

cheers -ben

<sharedLibString.zip><clang3.zip>

--
_,,,^..^,,,_
best, Eliot

_,,,^..^,,,_

best, Eliot

tblanchard

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

I'm mac osx

I will update the test library and ping back

On Mar 1, 2018, at 10:11 AM, Eliot Miranda <[hidden email]> wrote:

Todd,

On Thu, Mar 1, 2018 at 10:01 AM, Eliot Miranda <[hidden email]> wrote:
Hi Todd,

On Sun, Nov 26, 2017 at 8:24 AM, Todd Blanchard <[hidden email]> wrote:

i'm getting the idea that we should probably write a test suite/library for FFI

See the package FFI-Tests at http://source.squeak.org/FFI and the file sqFFITestFuncs.c

I suggest you extend these with an example that demonstrates your failing case. One thing to be very careful about is the platform. Are you on Mac OS X, linux or Win64? Mac OS X & linux use the SysV ABI (see e.g. https://software.intel.com/sites/default/files/article/402129/mpx-linux64-abi.pdf) whereas Win64 uses Microsoft's own ABI (see https://msdn.microsoft.com/en-us/library/ms235286.aspx?f=255&MSPPError=-2147217396). In particular under the Sys V ABI certain structs (structs having two 64-bit fields) will be passed in registers, if two registers are available at that point in the parameter list. The ThreadedX64SysVFFIPlugin is written to pass structs in this manner but there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion. Microsoft's own ABI passes structs of size 8, 16, 32, or 64 bits and __m64 in a single register. The ThreadedX64Win64FFIPlugin is written to pass structs in this manner but again there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion.

HTH

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:

On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:

I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)

I presume you meant...
  if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0)
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked

So it seems a similar problem exists outside our FFI.

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...

___microlang.h___
typedef unsigned uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);

___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{ CXSourceRange_ range = {0};
  range.ptr_data[0] = (void*)libraryString;
  return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{ // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
  if ((uintptr_t)range.ptr_data[0] & 0x1)
  { return 0; }
  else
  { return 1; }
}

___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString = "local_pointer";

void test( CXSourceRange_ range, char *note )
{ int result = clang_getRangeEnd_(range);
  if(result == 0)
  { printf("That failed (%s)\n", note); }
  else
  { printf("That worked (%s)\n", note); }
}

int main()
{ CXSourceRange_ range1 = clang_getArbitraryRange_();
  test(range1, "library string");

  CXSourceRange_ range2 = {0};
  range2.ptr_data[0] = (void*)localString;
  test(range2, "local string");
}

___Makefile___
default: clean static shared

clean:
  rm -f *so *App
  @echo

shared:
  clang -g -o libmicroclang.so -shared -fPIC microclang.c
  clang -g -o sharedApp test.c -L. -lmicroclang
  LD_LIBRARY_PATH=. ./sharedApp
  @echo

static:
  clang -g -o staticApp test.c microclang.c
  ./staticApp
  @echo

Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)

Further simplification dealing *only* with strings (see attached sharedLibString.zip)
(also attached is clang3.zip as a step along the way)

___microclang.c___
typedef unsigned uintptr_t;
const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString()
{ return myLibraryString;
}

int lib_testString( const char *aString )
{ unsigned test = (uintptr_t)aString & 0x1;
  printf("\n test=%d, aString-->%d\n", test, (uintptr_t)aString);
  if (test)
  { return 0; }
  else
  { return 1; }
}

___test.c___
#include <stdio.h>
int lib_testString( const char *aString );
const char *lib_getLibraryString();
const char *localString = "local_pointer";

void test( int result, char *note )
{ if(result == 0)
  { printf("That failed (%s)\n", note); }
  else
  { printf("That worked (%s)\n", note); }
}

int main()
{ const char * libraryString = lib_getLibraryString();
  test(lib_testString(libraryString), "library string");

  test(lib_testString(localString), "local string");
}

$ make > report

___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp

  test=0, aString-->4196150
  That worked (library string)

  test=0, aString-->4196068
  That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp

  test=1, aString-->-792512599
  That failed (library string)

  test=0, aString-->4196484
  That worked (local string)

cheers -ben

<sharedLibString.zip><clang3.zip>

--
_,,,^..^,,,_
best, Eliot

--
_,,,^..^,,,_
best, Eliot

tblanchard

Re: [Pharo-dev] FFI Struct Argument Pass By Value Fails on Mac 64 bit

Silly me.

I have no idea how to update this library. I thought it would be a file. I've used Monticello years ago in Squeak - however trying to load this repository into Pharo 6.1-64 is impossible. Invariably I end up with an undefined symbol and a couple of undismissable progress dialogs.

Also, its .c files that compile into a library, right? Where would such files be in this repository?

I love Smalltalk.

I'm really starting to hate this though.

On Mar 1, 2018, at 10:25 AM, todd blanchard <[hidden email]> wrote:

I'm mac osx

I will update the test library and ping back

On Mar 1, 2018, at 10:11 AM, Eliot Miranda <[hidden email]> wrote:

Todd,

On Thu, Mar 1, 2018 at 10:01 AM, Eliot Miranda <[hidden email]> wrote:
Hi Todd,

On Sun, Nov 26, 2017 at 8:24 AM, Todd Blanchard <[hidden email]> wrote:

i'm getting the idea that we should probably write a test suite/library for FFI

See the package FFI-Tests at http://source.squeak.org/FFI and the file sqFFITestFuncs.c

I suggest you extend these with an example that demonstrates your failing case. One thing to be very careful about is the platform. Are you on Mac OS X, linux or Win64? Mac OS X & linux use the SysV ABI (see e.g. https://software.intel.com/sites/default/files/article/402129/mpx-linux64-abi.pdf) whereas Win64 uses Microsoft's own ABI (see https://msdn.microsoft.com/en-us/library/ms235286.aspx?f=255&MSPPError=-2147217396). In particular under the Sys V ABI certain structs (structs having two 64-bit fields) will be passed in registers, if two registers are available at that point in the parameter list. The ThreadedX64SysVFFIPlugin is written to pass structs in this manner but there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion. Microsoft's own ABI passes structs of size 8, 16, 32, or 64 bits and __m64 in a single register. The ThreadedX64Win64FFIPlugin is written to pass structs in this manner but again there may be bugs in the code to identify precisely those structs that fulfill the register passing criterion.

HTH

On Nov 24, 2017, at 12:54 AM, Ben Coman <[hidden email]> wrote:

On 24 November 2017 at 13:16, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 21:59, Ben Coman <[hidden email]> wrote:

On 22 November 2017 at 13:38, Todd Blanchard <[hidden email]> wrote:

I've been trying to track this down for a couple weeks now.

I have concluded that structs passed by value to functions on the 64 bit VM are not properly populated. The struct's memory is all zero'd.

I found this while trying to work with LibClang and found that functions that fetched code locations from code ranges always returned invalid zero'd locations. After spending some time with lldb I have traced the problem into the native code and found that the argument is not correct.

I've carved out the wee bit of clang to reproduce this in a tiny library.

The gist of it is below and the entire file is included. Basically the struct passed to the function clang_getRangeStart is zero'd memory regardless of the data I send from the image side.

The build command I used on sierra is clang -shared -undefined dynamic_lookup -o microclang.dylib microclang.c

On Ubuntu 16.04 I used...
$ clang -shared -fPIC -o libmicroclang.so microclang.c

$ clang test.c -L. -l microclang
test.c:6:53: error: no member named 'begin_int_data' in 'CXSourceLocation'
if(clang_getRangeStart(clang_getArbitraryRange()).begin_int_data == 0)

I presume you meant...
  if(clang_getRangeStart(clang_getArbitraryRange()).int_data == 0)
so correcting and continuing...

$ clang test.c -L. -l microclang
$ LD_LIBRARY_PATH=. ./a.out
That failed

So I'm not sure how to proceed.
I was expecting that would work while Pharo failed.

Now interestingly...
$ clang test.c microlang.c
$ ./a.out
That worked

So it seems a similar problem exists outside our FFI.

cheers -ben

P.S. I refactored you code to extract a header file (attached)

The issue is still beyond my ken, but I've made some progress towards isolating/understanding the issue.
Attached zip exploded here for easy reference...

___microlang.h___
typedef unsigned uintptr_t;

typedef struct {
  const void *ptr_data[2];
} CXSourceRange_;

CXSourceRange_ clang_getArbitraryRange_();
int clang_getRangeEnd_(CXSourceRange_ range);

___microclang.c___
#include "microclang.h"
const char* libraryString = "library_pointer";

CXSourceRange_ clang_getArbitraryRange_()
{ CXSourceRange_ range = {0};
  range.ptr_data[0] = (void*)libraryString;
  return range;
}

int clang_getRangeEnd_(CXSourceRange_ range)
{ // Special decoding for CXSourceLocations for CXLoadedDiagnostics.
  if ((uintptr_t)range.ptr_data[0] & 0x1)
  { return 0; }
  else
  { return 1; }
}

___test.c___
#include <stdio.h>
#include "microclang.h"
const char* localString = "local_pointer";

void test( CXSourceRange_ range, char *note )
{ int result = clang_getRangeEnd_(range);
  if(result == 0)
  { printf("That failed (%s)\n", note); }
  else
  { printf("That worked (%s)\n", note); }
}

int main()
{ CXSourceRange_ range1 = clang_getArbitraryRange_();
  test(range1, "library string");

  CXSourceRange_ range2 = {0};
  range2.ptr_data[0] = (void*)localString;
  test(range2, "local string");
}

___Makefile___
default: clean static shared

clean:
  rm -f *so *App
  @echo

shared:
  clang -g -o libmicroclang.so -shared -fPIC microclang.c
  clang -g -o sharedApp test.c -L. -lmicroclang
  LD_LIBRARY_PATH=. ./sharedApp
  @echo

static:
  clang -g -o staticApp test.c microclang.c
  ./staticApp
  @echo

Now running...
$ make > report

gives...
___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp
That worked (library string)
That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp
That failed (library string)
That worked (local string)

Further simplification dealing *only* with strings (see attached sharedLibString.zip)
(also attached is clang3.zip as a step along the way)

___microclang.c___
typedef unsigned uintptr_t;
const char* myLibraryString = "library_pointer";

const char * lib_getLibraryString()
{ return myLibraryString;
}

int lib_testString( const char *aString )
{ unsigned test = (uintptr_t)aString & 0x1;
  printf("\n test=%d, aString-->%d\n", test, (uintptr_t)aString);
  if (test)
  { return 0; }
  else
  { return 1; }
}

___test.c___
#include <stdio.h>
int lib_testString( const char *aString );
const char *lib_getLibraryString();
const char *localString = "local_pointer";

void test( int result, char *note )
{ if(result == 0)
  { printf("That failed (%s)\n", note); }
  else
  { printf("That worked (%s)\n", note); }
}

int main()
{ const char * libraryString = lib_getLibraryString();
  test(lib_testString(libraryString), "library string");

  test(lib_testString(localString), "local string");
}

$ make > report

___report___
rm -f *so *App

clang -g -o staticApp test.c microclang.c
./staticApp

  test=0, aString-->4196150
  That worked (library string)

  test=0, aString-->4196068
  That worked (local string)

clang -g -o libmicroclang.so -shared -fPIC microclang.c
clang -g -o sharedApp test.c -L. -lmicroclang
LD_LIBRARY_PATH=. ./sharedApp

  test=1, aString-->-792512599
  That failed (library string)

  test=0, aString-->4196484
  That worked (local string)

cheers -ben

<sharedLibString.zip><clang3.zip>

--
_,,,^..^,,,_
best, Eliot

--
_,,,^..^,,,_
best, Eliot