DLLCC bad arguments

Hi all,

occasionally I get a "bad argument" execption with DLLCC when calling a function like this with a ByteArray that represents an UTF-8 encoded XML string:

        <C: short Function (unsigned char* xml) >

I really wonder what makes a pointer to a simple ByteArray become a "bad argument"? I tested with both char* and unsigned char*, to no avail. I would expect that the DLLCC machinery should not care about proper encoding and such, as bytes are just bytes. It should also not care about null termination, because that's the responsibility of the target C code, isn't it?

Inspecting the actual ByteArray passed as the argument, it seems to look just fine, although it can be rather big (1-3 MB).

What am I missing?

Andre


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

I've had the same problem in the past, and it also happened only with larger arrays (several MB).

The workaround was to write this:

        bytesPointer := byteArray copyToHeap.
        externalInterface myFunction: bytesPointer.
        bytesPointer free.

Instead of supplying the ByteArray itself as an argument, I supplied a CPointer to it and the problems dissapeared.

I suppose that internally the DLLCC machinery tries to make this copy itself but doesn't succeed in doing this because of a lack of memory or something like that. I assume that #copyToHeap can do it correctly because it may grow the VM's memory a bit if not enough heap space is available...

This is only my guess, I didn't investigate it in the VM, but it fixed the problem.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of andre
Sent: donderdag 17 november 2011 10:21
To: VWNC NC
Subject: [vwnc] DLLCC bad arguments

Hi all,

occasionally I get a "bad argument" execption with DLLCC when calling a function like this with a ByteArray that represents an UTF-8 encoded XML string:

        <C: short Function (unsigned char* xml) >

I really wonder what makes a pointer to a simple ByteArray become a "bad argument"? I tested with both char* and unsigned char*, to no avail. I would expect that the DLLCC machinery should not care about proper encoding and such, as bytes are just bytes. It should also not care about null termination, because that's the responsibility of the target C code, isn't it?

Inspecting the actual ByteArray passed as the argument, it seems to look just fine, although it can be rather big (1-3 MB).

What am I missing?

Andre


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

I'd have to add that it shouldn't be necessary to make a copy of the ByteArray when supplying it as an argument. If it were a threaded call, it would be needed because the memory could move around because of VWs memory management. For non threaded calls, I think just supplying the ByteArray should be enough.

In my case, it wasn't a problem to make copies of the byteArray. Performance was no issue here.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Mark Plas
Sent: donderdag 17 november 2011 11:00
To: andre; VWNC NC
Subject: Re: [vwnc] DLLCC bad arguments

I've had the same problem in the past, and it also happened only with larger arrays (several MB).

The workaround was to write this:

        bytesPointer := byteArray copyToHeap.
        externalInterface myFunction: bytesPointer.
        bytesPointer free.

Instead of supplying the ByteArray itself as an argument, I supplied a CPointer to it and the problems dissapeared.

I suppose that internally the DLLCC machinery tries to make this copy itself but doesn't succeed in doing this because of a lack of memory or something like that. I assume that #copyToHeap can do it correctly because it may grow the VM's memory a bit if not enough heap space is available...

This is only my guess, I didn't investigate it in the VM, but it fixed the problem.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of andre
Sent: donderdag 17 november 2011 10:21
To: VWNC NC
Subject: [vwnc] DLLCC bad arguments

Hi all,

occasionally I get a "bad argument" execption with DLLCC when calling a function like this with a ByteArray that represents an UTF-8 encoded XML string:

        <C: short Function (unsigned char* xml) >

I really wonder what makes a pointer to a simple ByteArray become a "bad argument"? I tested with both char* and unsigned char*, to no avail. I would expect that the DLLCC machinery should not care about proper encoding and such, as bytes are just bytes. It should also not care about null termination, because that's the responsibility of the target C code, isn't it?

Inspecting the actual ByteArray passed as the argument, it seems to look just fine, although it can be rather big (1-3 MB).

What am I missing?

Andre


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Am 17.11.2011 um 11:25 schrieb Mark Plas:

> I'd have to add that it shouldn't be necessary to make a copy of the ByteArray when supplying it as an argument. If it were a threaded call, it would be needed because the memory could move around because of VWs memory management. For non threaded calls, I think just supplying the ByteArray should be enough.
>
> In my case, it wasn't a problem to make copies of the byteArray. Performance was no issue here.


Thanks a lot, Mark. It seems to work fine. If I still encounter any issues, I will report back.

Performance is also not critical for me in this case, however it would be nice to not waste so much resources by copying large objects back and forth where not absolutely necessary.

Andre


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Andre wrote:
> occasionally I get a "bad argument" execption with DLLCC when calling
a
Maybe related to a DLLCC bug I reported in 2003. It was assigned AR
47137, but as far as I can see from the vw-dev-ann archive that has
never been corrected. The bug occurred when:
- parameter is declared as char * (LPCTSTR seems OK)
- argument is an MSCP1252String (e.g. anything read from a Windows file
is that class by default) rather than a ByteString, and is not copied to
heap etc.
- when the string length was a multiple of 4.

I can duplicate this still in 7.7.1a on Windows 7. E.g. change
GetDiskFreeSpaceEx:* to use char *, and pass a MSCP1252String name of a
4-char directory like 'D:\e' (that exists!). It returns nil; with a
ByteString it works.

More details below,
Steve

Product: VisualWorks 7.2
Platform: Windows XP, 98

When I call a 3rd party DLL function, if the string I pass
(MSCP1252String) as a 'char *' or 'const char *' argument is 40
characters long, the function apparently receives a different string.
Strings of length 35, 69, 66 and 67 work fine. Using optimizationLevel
#debug or #full has no effect.

My guess is that for MSCP1252Strings something's wrong with the
mechanism mentioned at the bottom of p.164 of the DLLCC manual:

"In the case of String arguments, a direct pointer to the Smalltalk
string is
passed if the string can be properly NULL terminated. If not, Smalltalk
makes a NULL terminated copy of the string in its object memory and a
pointer to the new copy is passed as the function's argument."

This seems not to work with a MSCP1252String whose size \\ 4 = 0. If I
put the string inline in the ExternalInterface function call, it's a
ByteString and works fine. (The string contains only characters
0-9a-zA-Z()& and space.) Similarly if I copy it inline ('1234' copy).
However, if I send it as a MSCP1252String, it fails:

MSCP1252String fromBytes: '1234' asByteArray

But if I create it as a MSCP1252String with one extra character, and
null out that character, it works:

bytes := '1234_' asByteArray.
bytes at: bytes size put: 0.
str := MSCP1252String fromBytes: bytes.

I've tested this for sizes 4, 16, 36, 40 - all behave the same.
Similarly 35, 37, 66, 67 and 69 all work fine with none of the above
fiddling. Another workaround is to send asByteString to MSCP1252String
before passing it, but according to the DLLCC manual, none of this
should be necessary: ByteEncodedStrings should be passable as-is (p71).

Steve

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Dear Steve,
    AR 47137 (Amend documentation regarding the meaning of char * and
null termination responsibilities) was fixed in candle jun10.3 (i.e. in
VW7.7.1).  As it's final title suggests, the fix was solely a change in
documentation.  (See also AR 43505, 'rejected' but only in the sense
that 47137 was the same issue.)

The conclusion of the AR was that "char *" does not equate to "pointer
to the first character of a null terminated string" and a proper fix
would require moving control of marshalling up to the image level (i.e.
a rewrite of DLLCC - not trivial).  Meanwhile, the application developer
is in charge of knowing whether the bytes starting at the memory pointed
at by a char * constitute a null terminated, somehow encoded string.

I quote the added documentation
------------
Moreover, note that you can implement string classes that are not
recognized as such by the virtual machine.  In those cases, then DLLCC
will not automatically null terminate the strings you pass as
arguments.  Since in general terms a C string is a pointer to a
character, it may not be possible for the virtual machine to correctly
detect every situation that requires null terminating a string.  In
fact, in some cases it may be better to avoid encoding since the virtual
machine may not be aware of the string's encoding (UTF8, UTF16, UCS2,
etc).  Therefore, in these cases, it may be better to encode a string
into a byte array following the conventions expected by the external
function to be called, and then invoke the external function passing a
pointer to the first byte in the byte array.  You can see examples of
this technique by examining senders of #encodeWide:.
-----------

In other words, it was at first thought of as a VM issue.  It was then
recognised as not solvable in that context, so the documentation was
changed to indicate the current behaviour.

(Info provided so those who've met this problem can comment on the
pattern we recommend and/or their perception of the importance of the
general fix.)

HTH
                Yours faithfully
                      Niall Ross


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Hi Niall,

Do you know why Andre's call (and mine) sometimes fails? He provides a ByteArray to a C function and it sometimes fails. In my case, for a given ByteArray, the call sometimes worked, and sometimes it didn't (for exactly the same contents).

Can you perhaps explain a bit why a ByteArray argument for a char* variable could be considered a bad argument?

Thanks,
Mark

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Niall Ross
Sent: donderdag 24 november 2011 16:36
To: Steven Kelly
Cc: VWNC NC; andre
Subject: Re: [vwnc] DLLCC bad arguments

Dear Steve,
    AR 47137 (Amend documentation regarding the meaning of char * and null termination responsibilities) was fixed in candle jun10.3 (i.e. in VW7.7.1).  As it's final title suggests, the fix was solely a change in documentation.  (See also AR 43505, 'rejected' but only in the sense that 47137 was the same issue.)

The conclusion of the AR was that "char *" does not equate to "pointer to the first character of a null terminated string" and a proper fix would require moving control of marshalling up to the image level (i.e.
a rewrite of DLLCC - not trivial).  Meanwhile, the application developer is in charge of knowing whether the bytes starting at the memory pointed at by a char * constitute a null terminated, somehow encoded string.

I quote the added documentation
------------
Moreover, note that you can implement string classes that are not recognized as such by the virtual machine.  In those cases, then DLLCC will not automatically null terminate the strings you pass as arguments.  Since in general terms a C string is a pointer to a character, it may not be possible for the virtual machine to correctly detect every situation that requires null terminating a string.  In fact, in some cases it may be better to avoid encoding since the virtual machine may not be aware of the string's encoding (UTF8, UTF16, UCS2, etc).  Therefore, in these cases, it may be better to encode a string into a byte array following the conventions expected by the external function to be called, and then invoke the external function passing a pointer to the first byte in the byte array.  You can see examples of this technique by examining senders of #encodeWide:.
-----------

In other words, it was at first thought of as a VM issue.  It was then recognised as not solvable in that context, so the documentation was changed to indicate the current behaviour.

(Info provided so those who've met this problem can comment on the pattern we recommend and/or their perception of the importance of the general fix.)

HTH
                Yours faithfully
                      Niall Ross


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

If the API you are calling are expecting a null terminated string, and your bytearray  doesn't contains a null terminator, you might experience such "random" behavior



--
Henrik Høyer
Chief Software Architect
[hidden email] * (+45) 4029 2092
Marievej 15 * 4600 Køge
www.sPeople.dk * (+45) 7023 7775


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Mark Plas
Sent: 24. november 2011 17:27
To: Niall Ross
Cc: VWNC NC
Subject: Re: [vwnc] DLLCC bad arguments

Hi Niall,

Do you know why Andre's call (and mine) sometimes fails? He provides a ByteArray to a C function and it sometimes fails. In my case, for a given ByteArray, the call sometimes worked, and sometimes it didn't (for exactly the same contents).

Can you perhaps explain a bit why a ByteArray argument for a char* variable could be considered a bad argument?

Thanks,
Mark

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Niall Ross
Sent: donderdag 24 november 2011 16:36
To: Steven Kelly
Cc: VWNC NC; andre
Subject: Re: [vwnc] DLLCC bad arguments

Dear Steve,
    AR 47137 (Amend documentation regarding the meaning of char * and null termination responsibilities) was fixed in candle jun10.3 (i.e. in VW7.7.1).  As it's final title suggests, the fix was solely a change in documentation.  (See also AR 43505, 'rejected' but only in the sense that 47137 was the same issue.)

The conclusion of the AR was that "char *" does not equate to "pointer to the first character of a null terminated string" and a proper fix would require moving control of marshalling up to the image level (i.e.
a rewrite of DLLCC - not trivial).  Meanwhile, the application developer is in charge of knowing whether the bytes starting at the memory pointed at by a char * constitute a null terminated, somehow encoded string.

I quote the added documentation
------------
Moreover, note that you can implement string classes that are not recognized as such by the virtual machine.  In those cases, then DLLCC will not automatically null terminate the strings you pass as arguments.  Since in general terms a C string is a pointer to a character, it may not be possible for the virtual machine to correctly detect every situation that requires null terminating a string.  In fact, in some cases it may be better to avoid encoding since the virtual machine may not be aware of the string's encoding (UTF8, UTF16, UCS2, etc).  Therefore, in these cases, it may be better to encode a string into a byte array following the conventions expected by the external function to be called, and then invoke the external function passing a pointer to the first byte in the byte array.  You can see examples of this technique by examining senders of #encodeWide:.
-----------

In other words, it was at first thought of as a VM issue.  It was then recognised as not solvable in that context, so the documentation was changed to indicate the current behaviour.

(Info provided so those who've met this problem can comment on the pattern we recommend and/or their perception of the importance of the general fix.)

HTH
                Yours faithfully
                      Niall Ross


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

On 24.11.2011 17:27, Mark Plas wrote:
> Hi Niall,
>
> Do you know why Andre's call (and mine) sometimes fails? He provides a ByteArray to a C function and it sometimes fails. In my case, for a given ByteArray, the call sometimes worked, and sometimes it didn't (for exactly the same contents).
>
> Can you perhaps explain a bit why a ByteArray argument for a char* variable could be considered a bad argument?
>
> Thanks,
> Mark
If the object space is anything like the one in Cog, objects are 4-byte
aligned.
So if your object with variable bytes have a size mod 4 != 0, there will
actually be 1-3 bytes 0-byte padding, and thus it acts as if "properly"
null-terminated.
If not however, you'll just keep on reading the next objects till a 0 is
reached
With no error checking, if you're lucky the call will return gibberish
results, if unlucky it crashes.
To me it sounds like WV does some sanity checking to see you are not
passing a "string" which extends outside the objects bounds, and return
an error instead.
(Would be interesting to see if it works for bytes representing a
non-terminated utf16 string though...)

As for properly encoding the argument, it _has_ to be the developers
responsibility, as the proper one is whatever the library you are
interfacing defines it as, and not something the marshalling code can
figure out (even if moved image-side).
For example, W-versions of the windows api expects utf16, non-W expects
the system character set, linux/mac libs usually expect utf8, etc...

Cheers,
Henry
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Thanks all for the answers.

In my case the argument wasn't a string, but an array of bytes describing the image to be used by a texture (for OpenGL). The argument expected by the function was "void *":

        void glTexImage2D(GLenum  target,  GLint  level,  GLint  internalFormat,  GLsizei  width,  GLsizei  height,  GLint  border,  GLenum  format,  GLenum  type,  const GLvoid *  data);

It's the last parameter "data" that caused the problems.

Since it's VW that returns the "Bad Argument" exception, it does sound like it's doing some sanity checking. It shouldn't be checking for 0 terminated strings in this case though. As the call sometimes succeeded, and sometimes not (with the same arguments), it could perhaps be related to checking for 4-byte or 8-byte alignment, though I wouldn't know how VW could know that it needs to perform such a check...

Maybe one of the VM engineers know what's going on?

Mark

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

> Since it's VW that returns the "Bad Argument" exception, it does sound like it's doing some sanity checking.

As far as I understood, VW needs to make a *copy* of the data to protect it from being garbage collected. That's why a missing 0 termination could be fatal.

The unpredictable results you get result from random memory contents located after your actual data.

IMHO it would make sense if VW could find other ways to ensure this protection without making a copy or moving data. Every C API that supports void* arguments also requires a size argument, if zero termination is not used.

I am passing several MB of data per call and wonder if this is any efficient given the above assumption.

Andre


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

In my case, since it's a ByteArray that is being supplied as an argument, there wouldn't be a need for a 0 termination byte to calculate the size, because ByteArrays know their size. It would even be wrong to calculate the size based on a 0 byte because the information in the ByteArray is about a bitmap. It can contain many zeroes.

The C-function being called will need to know how large the array is of course, but in my case this information has been supplied in other arguments. Still, the VM couldn't possibly know the meaning of the arguments so I don't think it would be able to decide that the argument is bad based on that. There's some other logic going on.

I don't think that making a copy on the heap is absolutely necessary. When it would be a threaded call, this could be needed, because the garbage collector could start moving stuff around while the external call is doing its work. But even then you could allocate the bytearray in FixedSpace and it would no longer be needed to make a copy of it. With a non-threaded call, the VM blocks, so a GC is not possible and as such it would be okay to supply a pointer to the ByteArray, as long as you know that the called function does not hold on to the pointer of course, because after the call finishes, memory might get moved or reclaimed. But this is also something that the VM couldn't know I think.

-----Original Message-----
From: andre [mailto:[hidden email]]
Sent: vrijdag 25 november 2011 11:03
To: Mark Plas
Cc: [hidden email]
Subject: Re: [vwnc] DLLCC bad arguments


> Since it's VW that returns the "Bad Argument" exception, it does sound like it's doing some sanity checking.

As far as I understood, VW needs to make a *copy* of the data to protect it from being garbage collected. That's why a missing 0 termination could be fatal.

The unpredictable results you get result from random memory contents located after your actual data.

IMHO it would make sense if VW could find other ways to ensure this protection without making a copy or moving data. Every C API that supports void* arguments also requires a size argument, if zero termination is not used.

I am passing several MB of data per call and wonder if this is any efficient given the above assumption.

Andre


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc