API parameter types

Parameters:

Return Type:

Could someone please provide the documentation for API parameter types?

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***

Also, please note that that material is still Copyright Cincom/ParcPlace-Digitalk, and any copy of it should be attributed to this.

Please note that there is a mistake that should probably be cleaned up.. It says for a ushort: "Four bytes are passed" and it should say "Two bytes are passed".

Jon



From:        Andreas Rosenberg <[hidden email]>
To:        [hidden email]
Date:        04/07/2011 08:12 AM
Subject:        Re: API parameter types
Sent by:        Using Visual Smalltalk for Windows/Enterprise <[hidden email]>

Thanks Derek!

I've combined the semantic documentation with the implementation details and
added both to our wiki site:

http://vse-wiki.apis.de/index.cgi/Documentation%20using%20Win32%20API
<http://vse-wiki.apis.de/index.cgi/Documentation%20using%20Win32%20API>

Regards
 Andreas


-----Original Message-----
From: Using Visual Smalltalk for Windows/Enterprise
[[hidden email]]On Behalf Of Derek Renouf
Sent: Donnerstag, 7. April 2011 02:04
To: [hidden email]
Subject: Re: API parameter types



Parameters:




boolean

Four bytes are passed. The object must be a Boolean. This corresponds to
type BOOL in C.


double

Eight bytes are passed. This type is intended for use with double-precision
floating-point arguments such as class Float.


handle

The first four bytes of the object are passed. The object can be a
variableByteSubclass and will usually be an ExternalHandle (or one of its
subclasses).long Four bytes are passed. The object may be an integer in the
range of -2147483648 .. +2147483647 (0x80000000 .. 0x7FFFFFFF), or it can be
a byte object, and the first four bytes are passed. This corresponds to type
LONG in C.


self

The object pointer itself is passed. This is used by user-defined DLL
functions.


short

Four bytes are passed. The object may be an integer in the range of -32768
.. +32767 (0x8000 .. 0x7FFF), or it can be a byte object, and the first two
bytes are passed. This corresponds to type SHORT in C.


struct

The four-byte address of the bytes of the object are passed. The object must
be a variableByteSubclass and will usually be an ExternalBuffer. This type
is used when the C argument is a pointer, such as a PSZ.


structIn

A struct pointer argument which is an IN parameter, i.e., the caller makes
no changes to the structure. This is a performance optimization of the
existing struct argument type.


structOut

A struct pointer argument which is an OUT parameter, i.e., the initial state
of the structure is not specified and the caller will set the values. This
is a performance optimization of the existing struct argument type.


structValue

This is a structure argument which is passed by value. For example, a POINTL
argument is an 8-byte structure passed by value on the call stack, in
contrast to a PPOINTL argument, which is a pointer to a POINTL structure.


ulong

Four bytes are passed. The object may be a positive integer the range of
0..4294967295 (0xFFFFFFFF), or it can be a byte object, and the first four
bytes are passed. This corresponds to type ULONG in C.


ushort

Four bytes are passed. The object may be a positive integer in the range of
0 to 65535 (0xFFFF), or it can be a byte object and the first two bytes are
passed. This corresponds to type USHORT in C.



Return Type:




hresult

A 32-bit return value of type HRESULT. This return type is used extensively
in OLE, but can also be used in non-OLE API's.


none

Is specified if the function does not return a value. This is equivalent to
declaring VOID in C.


short

A 2-byte Integer, signed


ushort

A 2-byte Integer, unsigned


boolean

A 4-byte Boolean


long

A 4-byte Integer, signed


ulong

A 4-byte String, representing an unsigned integer quantity


ulongReturn

A 4-byte Integer, unsigned


double

An 8-byte Float, signed


handle

A variableByteObject (String) from which an appropriate ExternalHandle can
be obtained via fromBytes: (Note: handle is actually the same as ulong)







From: Using Visual Smalltalk for Windows/Enterprise
[[hidden email]] On Behalf Of Leandro Caniglia
Sent: Wednesday, 6 April 2011 10:15 PM
To: [hidden email]
Subject: API parameter types



Could someone please provide the documentation for API parameter types?



Thanks!

Leandro



---

short

long

boolean

ushort

ulong

struct

handle

self

none

double

ulongReturn

selfIndirect

structValue

structIn

structOut

hresult





*** this signature added by listserv *** *** Visit
http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing
and VSWE-L membership management ***

*** this signature added by listserv *** *** Visit
http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing
and VSWE-L membership management ***


***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***

Some comments about the parameters.

Except for #structValue, all other parameters user 4 bytes of stack. For #structValue more than 4 bytes go on the stack, but it still has to be multiply of 4. On 32-bit Windows, the stack registers are always 4-aligned.

The OLE callback (and may be the normal callback) cannot handle the #structValue type. A solution is to manually split the value into 4-byte chunks and use those to reconstruct the structure. For example, if POINTL has to be used (8 bytes struct), divide it into X parameter of type #long (byte 0-3) and Y parameter of type #long (byte 4-7).

For the #struct type, the VM allocates memory and copies the binary contents of the object for that memory location. The VM then passes the address of this memory location to the API being called. Then the API returns, the contents of the memory is copied back into the object and the memory is freed.

The #structIn and #structOut or optimization of the #struct type to save one of the memory copy operations.

Depending on the size of the memory needed for #struct, #structIn or #structOut, the VM chooses the stack for small objects and heap allocation for larger objects. I am not sure about the size threshold, but my guess is about 4KB.

When calling the APIs, the usual way is to send #asParameter to each argument. This method in most cases returns self, but in some cases, for example for instances of ExternalBuffer, it will return the binary object that contains the buffer contents. The #asParameter method is nice convenience, but it has some drawbacks, because the programmer must know what is happening.

For #struct, #structIn or #structOut, the object being passed to the API must be a binary object. This has issues especially with ExternalBuffers that are not based on a Smalltalk ByteArray but on an ExternalAddress. ExternalAddress are usually passed with the type #ulong and with the helper method #asParameter. This means that the API may have to be declared twice; once with type #struct if the ExternalBuffer is created in Smalltalk, and once with type #ulong when it is based on ExternalAddress. Some logic is needed to determine which version of the API parameter is needed. Luckily, there is a trick / hack at least in newer VM’s that can overcome that annoyance. Declare the API parameter as #struct and don’t send #asParameter to the ExternalBuffer. Just pass the ExternalBuffer object directly to the API, even if it is a pointer object and therefore of the wrong type. The VM will extract the contents object out of the ExternalBuffer and then sense if the contents is a binary object or an external address and pass it to the API accordingly. I’ve seen this work for ExternalBuffer instance, but I am not sure how the VM detects if it is an ExternalBuffer.

The type #double is for 8-byte floating point. There is no #single, i.e. there is no support for 4-byte floats. A work around is to construct the binary data of the float yourself, i.e. construct a byte array of 4 bytes that contains the sign, exponent and significant and pass the data as #structValue. The OLE framework has helper methods to convert from 8-byte to 4-byte floats.

The #self type is usually used with user primitives. It is not used very often. However, imagine that you have an API that calculates hash of a binary object. The normal way is to use #struct (or #structIn) and pass the address of the binary data to the API. However, as discussed earlier, the VM will copy the data to a temporary memory before passing it to the API. To avoid the copy overhead, use the type #self, then the VM will pass the address of the real Smalltalk object. It is transparent to the API, but we’ve avoid the memory copy operation. Using #self in this way is only allowed if you are 100% sure the API call will block and not re-enter Smalltalk code somehow. If re-entered, a GC may move the object, and this is bad. Also, don’t use this to write to pointer objects, because writing to pointer objects requires the VM to be notified so it can update GC flags. I discourage using #self, but in few cases it may make sense.

To understand how the API parameters work, I recommend some playing. There is a very helpful API in the VM called ObjectAtAddress, but it can be misused to do other stuff. This API does very little. My guess is that C function for this will look like this:

OOP ObjectAtAddress(ULONG addr)

{

          return (OOP) addr;

}

In reality, it only does a cast from the address to Smalltalk object, and no math on anything else is performed.

VirtualMachineExe>> objectAtAddress: anAddress

    <api: ObjectAtAddress ulong self>

    ^self invalidArgument

“NB: The source code for #objectAtAddress: is property of Digital/Cincom “

But we can cheat and get it to do some tricks. For example:

addressOfObject: anObject

    <api: ObjectAtAddress self ulongReturn>

    ^self invalidArgument

In the last example, don’t rely on the address being constant. It may change after a GC. Few objects have fixed addresses (nil, true, false, characters and few others – see the USERPRIM sample in the documentation).

Some other examples:

addressOfStruct: anObject

    <api: ObjectAtAddress struct ulongReturn>

    ^self invalidArgument

longToBoolean: anObject

    <api: ObjectAtAddress long boolean>

    ^self invalidArgument

Just experiment with the API parameter type declarations to see how the VM converts API parameters.

Have fun!

-- Todor

Parameters:

Return Type:

Could someone please provide the documentation for API parameter types?

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***

Hi all,

I’d like to know how others work with large byte arrays. Of issue is that when you create certain sized ByteArray (or String for that matter), it takes a long period of time. For example a 1MB byte array will consistently take around ¼ of a second on my test machine (Core 2 – 2.5Ghz), using:

Time millisecondsToRun: [ ByteArray new: ( 1024 * 1024 ) ] à ~230ms

However,

Time millisecondsToRun: [ 3 timesRepeat: [ ByteArray new: ( 1024 * 256 ) ] ] à ~1ms

If you increase this to 4 times, the total time returns to the crazy amount of ¼ of a second (presumably due to the VM GC sweeping the universe?).

Time millisecondsToRun: [ 4 timesRepeat: [ ByteArray new: ( 1024 * 256 ) ] ]  à ~230ms

The issue is that creating a dozen of them or more takes seconds, which is not exactly what you’d expect from a modern PC.

Is there a way around this so that ByteArrays can be allocated more quickly?

I’ve considered using an ExternalAddress and just allocating memory, but this resource needs to be freed and this would require changes in code.

Time millisecondsToRun: [

      | mem |

      mem := ExternalAddress allocateMemory: ( 1024 * 1024 ).

      mem free.

]. à ~ 0 ms

-- Derek

This is most probably due to GC. Check if GC is happening during array creation. See Process gcCompactInterrupt / gcFlipInterrupt.

It should take about a millisec or so – no more. This is a cheap operation. But since the array is larger than 256KB, it is allocated directly in old-space. This may trigger GC of old space if not enough memory is available. GC of old space is expensive. Ensure that there is enough space there. Search the old postings for VirtualMachineConfiguration.

-- Todor

Hi all,

I’d like to know how others work with large byte arrays. Of issue is that when you create certain sized ByteArray (or String for that matter), it takes a long period of time. For example a 1MB byte array will consistently take around ¼ of a second on my test machine (Core 2 – 2.5Ghz), using:

Time millisecondsToRun: [ ByteArray new: ( 1024 * 1024 ) ] à ~230ms

However,

Time millisecondsToRun: [ 3 timesRepeat: [ ByteArray new: ( 1024 * 256 ) ] ] à ~1ms

If you increase this to 4 times, the total time returns to the crazy amount of ¼ of a second (presumably due to the VM GC sweeping the universe?).

Time millisecondsToRun: [ 4 timesRepeat: [ ByteArray new: ( 1024 * 256 ) ] ]  à ~230ms

The issue is that creating a dozen of them or more takes seconds, which is not exactly what you’d expect from a modern PC.

Is there a way around this so that ByteArrays can be allocated more quickly?

I’ve considered using an ExternalAddress and just allocating memory, but this resource needs to be freed and this would require changes in code.

Time millisecondsToRun: [

      | mem |

      mem := ExternalAddress allocateMemory: ( 1024 * 1024 ).

      mem free.

]. à ~ 0 ms

-- Derek

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***

Thanks Todor,

The dev image reports a total of 224MB memory used from Task Manager.

Looking at the result of VirtualMachineConfiguration class>>fromImage:

Old space bytes: 134,217,728

Arena bytes: 268,435,456

-- Derek

This is most probably due to GC. Check if GC is happening during array creation. See Process gcCompactInterrupt / gcFlipInterrupt.

It should take about a millisec or so – no more. This is a cheap operation. But since the array is larger than 256KB, it is allocated directly in old-space. This may trigger GC of old space if not enough memory is available. GC of old space is expensive. Ensure that there is enough space there. Search the old postings for VirtualMachineConfiguration.

-- Todor

Hi all,

I’d like to know how others work with large byte arrays. Of issue is that when you create certain sized ByteArray (or String for that matter), it takes a long period of time. For example a 1MB byte array will consistently take around ¼ of a second on my test machine (Core 2 – 2.5Ghz), using:

Time millisecondsToRun: [ ByteArray new: ( 1024 * 1024 ) ] à ~230ms

However,

Time millisecondsToRun: [ 3 timesRepeat: [ ByteArray new: ( 1024 * 256 ) ] ] à ~1ms

If you increase this to 4 times, the total time returns to the crazy amount of ¼ of a second (presumably due to the VM GC sweeping the universe?).

Time millisecondsToRun: [ 4 timesRepeat: [ ByteArray new: ( 1024 * 256 ) ] ]  à ~230ms

The issue is that creating a dozen of them or more takes seconds, which is not exactly what you’d expect from a modern PC.

Is there a way around this so that ByteArrays can be allocated more quickly?

I’ve considered using an ExternalAddress and just allocating memory, but this resource needs to be freed and this would require changes in code.

Time millisecondsToRun: [

      | mem |

      mem := ExternalAddress allocateMemory: ( 1024 * 1024 ).

      mem free.

]. à ~ 0 ms

-- Derek

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***

Also, if I restart the image so there is only 57MB reported usage from Task Manager, then the times are the same (or maybe 5-10ms faster).

Thanks Todor,

The dev image reports a total of 224MB memory used from Task Manager.

Looking at the result of VirtualMachineConfiguration class>>fromImage:

Old space bytes: 134,217,728

Arena bytes: 268,435,456

-- Derek

This is most probably due to GC. Check if GC is happening during array creation. See Process gcCompactInterrupt / gcFlipInterrupt.

It should take about a millisec or so – no more. This is a cheap operation. But since the array is larger than 256KB, it is allocated directly in old-space. This may trigger GC of old space if not enough memory is available. GC of old space is expensive. Ensure that there is enough space there. Search the old postings for VirtualMachineConfiguration.

-- Todor

Hi all,

I’d like to know how others work with large byte arrays. Of issue is that when you create certain sized ByteArray (or String for that matter), it takes a long period of time. For example a 1MB byte array will consistently take around ¼ of a second on my test machine (Core 2 – 2.5Ghz), using:

Time millisecondsToRun: [ ByteArray new: ( 1024 * 1024 ) ] à ~230ms

However,

Time millisecondsToRun: [ 3 timesRepeat: [ ByteArray new: ( 1024 * 256 ) ] ] à ~1ms

If you increase this to 4 times, the total time returns to the crazy amount of ¼ of a second (presumably due to the VM GC sweeping the universe?).

Time millisecondsToRun: [ 4 timesRepeat: [ ByteArray new: ( 1024 * 256 ) ] ]  à ~230ms

The issue is that creating a dozen of them or more takes seconds, which is not exactly what you’d expect from a modern PC.

Is there a way around this so that ByteArrays can be allocated more quickly?

I’ve considered using an ExternalAddress and just allocating memory, but this resource needs to be freed and this would require changes in code.

Time millisecondsToRun: [

      | mem |

      mem := ExternalAddress allocateMemory: ( 1024 * 1024 ).

      mem free.

]. à ~ 0 ms

-- Derek

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***

GC of old space is triggered entirely from Smalltalk code. See senders of #unusedMemory. You can play with the parameters of ProcessScheduler, especially OldSpaceThresholdIncrement, and find a strategy that is appropriate for your usage of such large byte arrays. As Todor said, they will always be allocated in old space, so even if they are only needed for a short time, you will eventually have to perform an old space GC to get rid of them.

b.t.w. – large byte arrays and api calls of type struct (as an extension to the api parameter discussion): when passing such data, the vm copies them to the heap, and after the api call returns, the heap is freed. Unless you do this in a tight loop, the time for this is probably not worth to worry about; but we found one problem with this behavior, namely that the action of clearing the heap also clears the GetLastError() state of the thread. We sometimes encountered the situation that an api call returns false, and then the typical call to self osError just returns 0. This problem vanishes when you change the parameter type to #ulong, and pass the external address of the byte array #copyToOSMemory. Of course with the drawback (as you said) that this changes the api, you have to take care to free the memory for yourself.

Manfred

This is most probably due to GC. Check if GC is happening during array creation. See Process gcCompactInterrupt / gcFlipInterrupt.

It should take about a millisec or so – no more. This is a cheap operation. But since the array is larger than 256KB, it is allocated directly in old-space. This may trigger GC of old space if not enough memory is available. GC of old space is expensive. Ensure that there is enough space there. Search the old postings for VirtualMachineConfiguration.

-- Todor

Hi all,

I’d like to know how others work with large byte arrays. Of issue is that when you create certain sized ByteArray (or String for that matter), it takes a long period of time. For example a 1MB byte array will consistently take around ¼ of a second on my test machine (Core 2 – 2.5Ghz), using:

Time millisecondsToRun: [ ByteArray new: ( 1024 * 1024 ) ] à ~230ms

However,

Time millisecondsToRun: [ 3 timesRepeat: [ ByteArray new: ( 1024 * 256 ) ] ] à ~1ms

If you increase this to 4 times, the total time returns to the crazy amount of ¼ of a second (presumably due to the VM GC sweeping the universe?).

Time millisecondsToRun: [ 4 timesRepeat: [ ByteArray new: ( 1024 * 256 ) ] ]  à ~230ms

The issue is that creating a dozen of them or more takes seconds, which is not exactly what you’d expect from a modern PC.

Is there a way around this so that ByteArrays can be allocated more quickly?

I’ve considered using an ExternalAddress and just allocating memory, but this resource needs to be freed and this would require changes in code.

Time millisecondsToRun: [

      | mem |

      mem := ExternalAddress allocateMemory: ( 1024 * 1024 ).

      mem free.

]. à ~ 0 ms

-- Derek

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***