Bug in OSPtr #address and friends or is it just my lack of C knowledge?

I'm working on making a Smalltalk app to interface with LibUSB.  It has a function "libusb_get_device_list" that sets a pointer to what I think is an array of pointers to USB Device Descriptor structures.  I'm having trouble with this so, maybe I will ask a question about that later.  For now, when I try to look at the OSPtr (maybe I'm using the wrong class here) returned by libusb_get_device_list and sent it the #address message it blows up because #address sends "+" to reference that is a ByteArray and doesn't understand "+".

Am I doing something wrong (very, very possible) or is this a bug.

Lou

Thank you very much for all the help.  Your example has gotten me past my problem with these pointers to pointers.

The init of LibUSB is needed (at least once).

I still think something is odd with #address and #asInteger and maybe others as they send "+" to #reference (a ByteArray).  Now, it seems #reference can be an Integer and when it is a ByteArray, one wouldn't (and shouldn't) call #address and friends.

Lou

On Saturday, January 30, 2016 at 5:34:42 AM UTC-5, Marten Feldtmann wrote:

| pfInit pfDeviceList numberOfDevices |
pfDeviceList := PlatformFunction fromArray: #('C' 'libusb_get_device_list' nil 'libusb-1.0.dll' #(#pointer #pointer) #uint32).

"one pointer to a possible array of additional OSObjectPointer"
osPtr := OSObjectPointer itemType: OSObjectPointer.
numberOfDevices := pfDeviceList callWith: OSHwnd null with: osPtr.

osPtrArray :=  osPtr at: 0.
Transcript cr ; show: numberOfDevices asString.

"The field is actually one entry larger - the last one is a NULL entry"
0 to: numberOfDevices do: [ :anIndex |
    Transcript cr ; show: anIndex asString ; show: ' - ' ; show: (osPtrArray at: anIndex)  asString
]

Now (osPtrArray at: x) return OSVoid pointers, which are pretty useful in further calls.

The #address message can be a little confusing because of what an OSObject's reference field can represent.

1. OSPtr pointing to OS native memory (Outside the Smalltalk managed heap memory).

    - reference is an integer....this integer is the native memory address that the OSPtr refers to.
    - Use the class side #calloc based APIs to allocate native memory.
    - For example, OSInt32 calloc will create a new pointer to a 32-bit signed integer...in C something like int * p = (int *)calloc(sizeof(int));
    - For the ones you calloc'd....then you are responsible for freeing them.

    - I use these about 95% of the time.

2. OSPtr pointing to Smalltalk managed memory

   - reference is a ByteArray smalltalk object....address doesn't really make sense because ByteArrays *can* move due to Garbage collection, unless fixed, and therefore it's actual address would depend on when you asked.

   - Use the class side #new based APIs to allocate Smalltalk Memory

   - Because you are referring to Smalltalk memory....when you call 'free' on these...the vm will simply nil out the reference field....the memory will actually be freed by the GC at some later point

3. Immediate objects.  Well this isn't really OSPtr related...but is OSObject related and therefore the 'reference' field has yet another meaning.

  - reference is still an integer...but it represents an actual value...and should not be interpreted as a memory location.  For this reason, there is a separate non-OSPtr related subclass of OSObject called OSImmediate to handle these.

So for better or worse...this is why you see the comment in OSPtr>>address that says

On Saturday, January 30, 2016 at 9:07:27 AM UTC-5, Louis LaBrunda wrote:

Hi Marten,

Thank you very much for all the help.  Your example has gotten me past my problem with these pointers to pointers.

The init of LibUSB is needed (at least once).

I still think something is odd with #address and #asInteger and maybe others as they send "+" to #reference (a ByteArray).  Now, it seems #reference can be an Integer and when it is a ByteArray, one wouldn't (and shouldn't) call #address and friends.

Lou

On Saturday, January 30, 2016 at 5:34:42 AM UTC-5, Marten Feldtmann wrote:

| pfInit pfDeviceList numberOfDevices |
pfDeviceList := PlatformFunction fromArray: #('C' 'libusb_get_device_list' nil 'libusb-1.0.dll' #(#pointer #pointer) #uint32).

"one pointer to a possible array of additional OSObjectPointer"
osPtr := OSObjectPointer itemType: OSObjectPointer.
numberOfDevices := pfDeviceList callWith: OSHwnd null with: osPtr.

osPtrArray :=  osPtr at: 0.
Transcript cr ; show: numberOfDevices asString.

"The field is actually one entry larger - the last one is a NULL entry"
0 to: numberOfDevices do: [ :anIndex |
    Transcript cr ; show: anIndex asString ; show: ' - ' ; show: (osPtrArray at: anIndex)  asString
]

Now (osPtrArray at: x) return OSVoid pointers, which are pretty useful in further calls.

Thanks for the explanation.  It is very helpful.  I was starting to think that the problem with sending #+ to a ByteArray wasn't real, because proper use of OSPtr and its subclasses would not result in #+ being sent to a ByteArray.  My lack of knowledge of C and the OSPtr classes and how they relate to their C counterparts lead me to just throw stuff against the wall and see what happens.  Sometimes things went boom and confused me even more.

Thanks to Marten I'm back on track.  I think and hope that in the end I will have a VA Smalltalk interface to LibUSB that is worth sharing.

Lou

On Saturday, January 30, 2016 at 9:41:48 PM UTC-5, Seth Berman wrote:

Hello Lou,

The #address message can be a little confusing because of what an OSObject's reference field can represent.

1. OSPtr pointing to OS native memory (Outside the Smalltalk managed heap memory).

    - reference is an integer....this integer is the native memory address that the OSPtr refers to.
    - Use the class side #calloc based APIs to allocate native memory.
    - For example, OSInt32 calloc will create a new pointer to a 32-bit signed integer...in C something like int * p = (int *)calloc(sizeof(int));
    - For the ones you calloc'd....then you are responsible for freeing them.

    - I use these about 95% of the time.

2. OSPtr pointing to Smalltalk managed memory

   - reference is a ByteArray smalltalk object....address doesn't really make sense because ByteArrays *can* move due to Garbage collection, unless fixed, and therefore it's actual address would depend on when you asked.

   - Use the class side #new based APIs to allocate Smalltalk Memory

   - Because you are referring to Smalltalk memory....when you call 'free' on these...the vm will simply nil out the reference field....the memory will actually be freed by the GC at some later point

3. Immediate objects.  Well this isn't really OSPtr related...but is OSObject related and therefore the 'reference' field has yet another meaning.

  - reference is still an integer...but it represents an actual value...and should not be interpreted as a memory location.  For this reason, there is a separate non-OSPtr related subclass of OSObject called OSImmediate to handle these.

So for better or worse...this is why you see the comment in OSPtr>>address that says

         "This message is invalid for immediate (non-pointer) objects

and for objects whose data element is stored in Smalltalk memory"

From my list...it says use this for use case number 1 only.

Hope this helps,

-- Seth

On Saturday, January 30, 2016 at 9:07:27 AM UTC-5, Louis LaBrunda wrote:

Hi Marten,

Thank you very much for all the help.  Your example has gotten me past my problem with these pointers to pointers.

The init of LibUSB is needed (at least once).

I still think something is odd with #address and #asInteger and maybe others as they send "+" to #reference (a ByteArray).  Now, it seems #reference can be an Integer and when it is a ByteArray, one wouldn't (and shouldn't) call #address and friends.

Lou

On Saturday, January 30, 2016 at 5:34:42 AM UTC-5, Marten Feldtmann wrote:

| pfInit pfDeviceList numberOfDevices |
pfDeviceList := PlatformFunction fromArray: #('C' 'libusb_get_device_list' nil 'libusb-1.0.dll' #(#pointer #pointer) #uint32).

"one pointer to a possible array of additional OSObjectPointer"
osPtr := OSObjectPointer itemType: OSObjectPointer.
numberOfDevices := pfDeviceList callWith: OSHwnd null with: osPtr.

osPtrArray :=  osPtr at: 0.
Transcript cr ; show: numberOfDevices asString.

"The field is actually one entry larger - the last one is a NULL entry"
0 to: numberOfDevices do: [ :anIndex |
    Transcript cr ; show: anIndex asString ; show: ' - ' ; show: (osPtrArray at: anIndex)  asString
]

Now (osPtrArray at: x) return OSVoid pointers, which are pretty useful in further calls.

I think this library will provide a neat capability.  Looks like they provide a 64-bit version too...I would be happy to test it on our internal 64-bit vm when you get to that point.  Just let me know.

-- Seth

On Monday, February 1, 2016 at 8:44:25 AM UTC-5, Louis LaBrunda wrote:

Hi Seth,

Thanks for the explanation.  It is very helpful.  I was starting to think that the problem with sending #+ to a ByteArray wasn't real, because proper use of OSPtr and its subclasses would not result in #+ being sent to a ByteArray.  My lack of knowledge of C and the OSPtr classes and how they relate to their C counterparts lead me to just throw stuff against the wall and see what happens.  Sometimes things went boom and confused me even more.

Thanks to Marten I'm back on track.  I think and hope that in the end I will have a VA Smalltalk interface to LibUSB that is worth sharing.

Lou

On Saturday, January 30, 2016 at 9:41:48 PM UTC-5, Seth Berman wrote:

Hello Lou,

The #address message can be a little confusing because of what an OSObject's reference field can represent.

1. OSPtr pointing to OS native memory (Outside the Smalltalk managed heap memory).

    - reference is an integer....this integer is the native memory address that the OSPtr refers to.
    - Use the class side #calloc based APIs to allocate native memory.
    - For example, OSInt32 calloc will create a new pointer to a 32-bit signed integer...in C something like int * p = (int *)calloc(sizeof(int));
    - For the ones you calloc'd....then you are responsible for freeing them.

    - I use these about 95% of the time.

2. OSPtr pointing to Smalltalk managed memory

   - reference is a ByteArray smalltalk object....address doesn't really make sense because ByteArrays *can* move due to Garbage collection, unless fixed, and therefore it's actual address would depend on when you asked.

   - Use the class side #new based APIs to allocate Smalltalk Memory

   - Because you are referring to Smalltalk memory....when you call 'free' on these...the vm will simply nil out the reference field....the memory will actually be freed by the GC at some later point

3. Immediate objects.  Well this isn't really OSPtr related...but is OSObject related and therefore the 'reference' field has yet another meaning.

  - reference is still an integer...but it represents an actual value...and should not be interpreted as a memory location.  For this reason, there is a separate non-OSPtr related subclass of OSObject called OSImmediate to handle these.

So for better or worse...this is why you see the comment in OSPtr>>address that says

         "This message is invalid for immediate (non-pointer) objects

and for objects whose data element is stored in Smalltalk memory"

From my list...it says use this for use case number 1 only.

Hope this helps,

-- Seth

On Saturday, January 30, 2016 at 9:07:27 AM UTC-5, Louis LaBrunda wrote:

Hi Marten,

Thank you very much for all the help.  Your example has gotten me past my problem with these pointers to pointers.

The init of LibUSB is needed (at least once).

I still think something is odd with #address and #asInteger and maybe others as they send "+" to #reference (a ByteArray).  Now, it seems #reference can be an Integer and when it is a ByteArray, one wouldn't (and shouldn't) call #address and friends.

Lou

On Saturday, January 30, 2016 at 5:34:42 AM UTC-5, Marten Feldtmann wrote:

| pfInit pfDeviceList numberOfDevices |
pfDeviceList := PlatformFunction fromArray: #('C' 'libusb_get_device_list' nil 'libusb-1.0.dll' #(#pointer #pointer) #uint32).

"one pointer to a possible array of additional OSObjectPointer"
osPtr := OSObjectPointer itemType: OSObjectPointer.
numberOfDevices := pfDeviceList callWith: OSHwnd null with: osPtr.

osPtrArray :=  osPtr at: 0.
Transcript cr ; show: numberOfDevices asString.

"The field is actually one entry larger - the last one is a NULL entry"
0 to: numberOfDevices do: [ :anIndex |
    Transcript cr ; show: anIndex asString ; show: ' - ' ; show: (osPtrArray at: anIndex)  asString
]

Now (osPtrArray at: x) return OSVoid pointers, which are pretty useful in further calls.

Hi Seth,

Thanks for the offer, I will take you up on it.  I have about 100 functions defined in a Pragma and about 20 structures defined.  I have tested and have working LibUsbErrorName, LibUsbGetDeviceList, LibUsbGetDeviceDescriptor, LibUsbStrError, LibUsbGetVersion and some smalltalk methods that make calling the LibUsb functions easier.

I think most of the functions defined in the Pragma are correct.  I'm not sure about those that take structures as input or that fill in structures.  I'm less confident about the definition of the structures.  Especially those that have structures in the structures.  I also find different documentation for some structures, some are longer than others.  I'm not sure how to handle that.

I'm more than willing to do the work but it would be a big help if someone could check what I have so far.  And give me some help where my knowledge of C falls short.

Lou

I'd be happy to help...send it along.

-- Seth

On Thursday, February 4, 2016 at 4:13:28 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

Thanks for the offer, I will take you up on it.  I have about 100 functions defined in a Pragma and about 20 structures defined.  I have tested and have working LibUsbErrorName, LibUsbGetDeviceList, LibUsbGetDeviceDescriptor, LibUsbStrError, LibUsbGetVersion and some smalltalk methods that make calling the LibUsb functions easier.

I think most of the functions defined in the Pragma are correct.  I'm not sure about those that take structures as input or that fill in structures.  I'm less confident about the definition of the structures.  Especially those that have structures in the structures.  I also find different documentation for some structures, some are longer than others.  I'm not sure how to handle that.

I'm more than willing to do the work but it would be a big help if someone could check what I have so far.  And give me some help where my knowledge of C falls short.

Lou

Hey Seth,

That's great!!  Attached is version 0.01 of what I have so far.  Kernel is the only prerequisite.  I would like to end up with a package that works with both Windows and Linux.

You will need to get a copy of the LibUSB dll (I'm using libusb-1.0.dll that I think I got here: project libusb).  In the Pragma I name the library LibUSB.  I put the dll in the folder with my test image and point to it with: LibUSB=libusb-1.0.dll in the [PlatformLibrary Name Mappings] section if the Ini file.

If you execute "KscLibUsbApp getUsbDeviceDescriptors inspect" if it works you will get an array of device descriptors.  Lately I have been having problems with libusb_init.  I will keep playing and send another post when I figure things out or give up and ask you a question (if I can think of the right one).  For now I want to get this posted. 

All of the classes start with "KscLibUsb", I'm open to changing that by maybe removing the "Ksc" part if you like.

You are of course free to help in any way you like.  I think going over all the function definitions in the Pragma would be a good place to start.  I used the definitions found here: functions, you can use those or any you like.

Do you know if the 64 bit version keeps the function names or add something like "64" to the end?  I ask because I expect the Pragma definitions would be different, with different names we could just put both in the same Pragma, if the names are the same we will need to think of something else, like separate packages.

If you want to make changes yourself, we will have to come up with some way to merge things.  I'm open to suggestions.

Lou

Thanks, I've imported and loaded it.

I new I would have to ask these questions...so I will now.

Do you want this to be compatible with < 8.6.2 images?

If so, then we must assume we don't have access to the new OSStructure capabilities in 8.6.2 which compute fixedSize, offsets (based on per-platform alignment rules and bitness) for you and permit the instance-size accessors to be reduced to symbolic access instead of hard-coded offsets.

The downside is

1. You would need 8.6.2 or above.

The upside is

1. You don't need separate 32/64-bit definitions or conditionals anywhere.
2. You don't need to hand compute fixedSize or any of the offsets
3. Platform-specific ABI rules are baked into the offset calculator...so you don't need to handle any adjustments to offsets based on platform-specific alignment rule

On Friday, February 5, 2016 at 12:25:42 PM UTC-5, Louis LaBrunda wrote:

Hey Seth,

That's great!!  Attached is version 0.01 of what I have so far.  Kernel is the only prerequisite.  I would like to end up with a package that works with both Windows and Linux.

You will need to get a copy of the LibUSB dll (I'm using libusb-1.0.dll that I think I got here: <a href="http://sourceforge.net/projects/libusb/files/libusb-1.0/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;">project libusb).  In the Pragma I name the library LibUSB.  I put the dll in the folder with my test image and point to it with: LibUSB=libusb-1.0.dll in the [PlatformLibrary Name Mappings] section if the Ini file.

If you execute "KscLibUsbApp getUsbDeviceDescriptors inspect" if it works you will get an array of device descriptors.  Lately I have been having problems with libusb_init.  I will keep playing and send another post when I figure things out or give up and ask you a question (if I can think of the right one).  For now I want to get this posted. 

All of the classes start with "KscLibUsb", I'm open to changing that by maybe removing the "Ksc" part if you like.

You are of course free to help in any way you like.  I think going over all the function definitions in the Pragma would be a good place to start.  I used the definitions found here: <a href="http://libusb.sourceforge.net/api-1.0/api.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;">functions, you can use those or any you like.

Do you know if the 64 bit version keeps the function names or add something like "64" to the end?  I ask because I expect the Pragma definitions would be different, with different names we could just put both in the same Pragma, if the names are the same we will need to think of something else, like separate packages.

If you want to make changes yourself, we will have to come up with some way to merge things.  I'm open to suggestions.

Lou

On Thursday, February 4, 2016 at 4:24:43 PM UTC-5, Seth Berman wrote:

Hi Lou,

I'd be happy to help...send it along.

-- Seth

On Thursday, February 4, 2016 at 4:13:28 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

Thanks for the offer, I will take you up on it.  I have about 100 functions defined in a Pragma and about 20 structures defined.  I have tested and have working LibUsbErrorName, LibUsbGetDeviceList, LibUsbGetDeviceDescriptor, LibUsbStrError, LibUsbGetVersion and some smalltalk methods that make calling the LibUsb functions easier.

I think most of the functions defined in the Pragma are correct.  I'm not sure about those that take structures as input or that fill in structures.  I'm less confident about the definition of the structures.  Especially those that have structures in the structures.  I also find different documentation for some structures, some are longer than others.  I'm not sure how to handle that.

I'm more than willing to do the work but it would be a big help if someone could check what I have so far.  And give me some help where my knowledge of C falls short.

Lou

If you have 8.6.2 image, an example of the new OSStructure capability is in MZStream (and basically everywhere else that was impacted by 64-bit).
On the class side...you see initializeAfterLoad.  However, you see that fixedSize is mysteriously not there.  Instead, you see the member names as normal, but this is followed by data type descriptors for each member name.

Notice the large number of #pointer fields.  This would be incredibly tedious to do by hand and cover 32/64 bit...and so it's not.  You just provide one definition, and then on the instance side you present the member name instead of the offset (because the offset would be different in 32/64-bit).
We technically have the information to figure out the datatype in the accessor on the instance side...but we chose not to do this because it's an extra level of indirection in what is a performance-critical area.  In the future we may provide that capability.

This framework also facilitates embedded or anonymous structures...one of the datatypes specified on #initializeOnLoad can be the name of another OSStructure....in which case it handles all offset/alignment/size computations.

There is some updated documentation for 8.6.2 in the Programmers Reference -> VA Smalltalk Virtual Machine API -> OSObject -> OSStructure section to describe this.
You can take a look here: http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/stpr451.html

-- Seth

On Friday, February 5, 2016 at 12:51:29 PM UTC-5, Seth Berman wrote:

Hello Lou,

Thanks, I've imported and loaded it.

I new I would have to ask these questions...so I will now.

Do you want this to be compatible with < 8.6.2 images?

If so, then we must assume we don't have access to the new OSStructure capabilities in 8.6.2 which compute fixedSize, offsets (based on per-platform alignment rules and bitness) for you and permit the instance-size accessors to be reduced to symbolic access instead of hard-coded offsets.

The downside is

1. You would need 8.6.2 or above.

The upside is

1. You don't need separate 32/64-bit definitions or conditionals anywhere.
2. You don't need to hand compute fixedSize or any of the offsets
3. Platform-specific ABI rules are baked into the offset calculator...so you don't need to handle any adjustments to offsets based on platform-specific alignment rule

I'll leave that up to you...some of the defined structs you have are considered opaque from the header file..so you don't really need to model them as OSStructures...they can just be void * as far as your concerned.

The only structures I see that are impacted by 64-bit would be the ones that have pointers in them.  This changes the offsets for any field after that.  The platform alignment rules should probably be ok...these are typically

for double datatypes or the first structure field on AIX...and I don't see, at first glance, any impact in that area.

Let me know how you would like to proceed and I'll try to make any adjustments accordingly.

-- Seth

On Friday, February 5, 2016 at 12:25:42 PM UTC-5, Louis LaBrunda wrote:

Hey Seth,

That's great!!  Attached is version 0.01 of what I have so far.  Kernel is the only prerequisite.  I would like to end up with a package that works with both Windows and Linux.

You will need to get a copy of the LibUSB dll (I'm using libusb-1.0.dll that I think I got here: <a href="http://sourceforge.net/projects/libusb/files/libusb-1.0/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;">project libusb).  In the Pragma I name the library LibUSB.  I put the dll in the folder with my test image and point to it with: LibUSB=libusb-1.0.dll in the [PlatformLibrary Name Mappings] section if the Ini file.

If you execute "KscLibUsbApp getUsbDeviceDescriptors inspect" if it works you will get an array of device descriptors.  Lately I have been having problems with libusb_init.  I will keep playing and send another post when I figure things out or give up and ask you a question (if I can think of the right one).  For now I want to get this posted. 

All of the classes start with "KscLibUsb", I'm open to changing that by maybe removing the "Ksc" part if you like.

You are of course free to help in any way you like.  I think going over all the function definitions in the Pragma would be a good place to start.  I used the definitions found here: <a href="http://libusb.sourceforge.net/api-1.0/api.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;">functions, you can use those or any you like.

Do you know if the 64 bit version keeps the function names or add something like "64" to the end?  I ask because I expect the Pragma definitions would be different, with different names we could just put both in the same Pragma, if the names are the same we will need to think of something else, like separate packages.

If you want to make changes yourself, we will have to come up with some way to merge things.  I'm open to suggestions.

Lou

On Thursday, February 4, 2016 at 4:24:43 PM UTC-5, Seth Berman wrote:

Hi Lou,

I'd be happy to help...send it along.

-- Seth

On Thursday, February 4, 2016 at 4:13:28 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

Thanks for the offer, I will take you up on it.  I have about 100 functions defined in a Pragma and about 20 structures defined.  I have tested and have working LibUsbErrorName, LibUsbGetDeviceList, LibUsbGetDeviceDescriptor, LibUsbStrError, LibUsbGetVersion and some smalltalk methods that make calling the LibUsb functions easier.

I think most of the functions defined in the Pragma are correct.  I'm not sure about those that take structures as input or that fill in structures.  I'm less confident about the definition of the structures.  Especially those that have structures in the structures.  I also find different documentation for some structures, some are longer than others.  I'm not sure how to handle that.

I'm more than willing to do the work but it would be a big help if someone could check what I have so far.  And give me some help where my knowledge of C falls short.

Lou

Hi Seth,

I have v8.6.2 but haven't installed it yet.  Do you know if there are any changes to the DLLs and VM that come with v8.6.1 and v8.6.2?  I ask because if there are no differences I will install v8.6.2 and use it for all my other work.  When there are changes to our product, I just deliver a new image.  The DLLs and VM are not replaced at the user site.  If they need to be, things get more complicated.

That said I would like to move to v8.6.2 and if that makes our life easier for this LibUSB project, all the better.  If you get back to me soon on this, I will see about moving to v8.6.2.

Lou

There are some changes to DLLs in 8.6.2...the ones that would impact deployment are the changes to OpenSSL.

Those are referenced here in the migration guide: http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8617.html

There are new envy library primitives for 8.6.2 for development images.  It probably doesn't impact your deployment, but for your development use with the ENVY repository...it requires some understanding.
Those are referenced here in the migration guide: http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8612.html

In short, the code you develop using the new library primitives (shipped with >= 8.6.2) will NOT be properly readable by images using the older library primitives ( < 8.6.2).
If your running a mixed environment, then the new library primitives must be copied to older versions on the product to use so they can properly decompress the >= 8.6.2 source code.

The importance of keeping as current as possible comes with your desire to use the 64-bit vm when it is released.  The 64-bit vm will definitely NOT run images older than 8.6.2 and never will.

Any third-party native library code not developed by us, and that would be impacted by 64-bit, would also need to be updated.  This is unavoidable, as the 32 to 64-bit image translator can not possibly have

knowledge of how some arbitrary 3rd party DLL is impacted by 64-bit and magically update the OSObjects.  This is why it is preferential to switch to the new OSObject capabilities sooner, rather than later, if you

have interest in running the 64-bit vm.  It reconfigures itself on startup based on the bitness so you only have 1 definition of an OSObject to support both 32 and 64-bit.

There is no doubt that the new capability makes life easier in OSObject development because hand computing anything no longer necessary....but as I said...the unavoidable cost is version compatibility.

The old way of doing things is still supported in 8.6.2 and beyond...if you stick to this it just means that things will have the potential to get messy when you try to port it to 64-bit when you must hand-compute everything and provide different definitions.

Some native libraries may have no issue (though very rare).  If some theoretical library had the same exact definitions/offsets/alignments/packing.....in both 32/64-bit for everything....then nothing would need to change.  However, I wouldn't count on this being the case:)

On Friday, February 5, 2016 at 1:19:36 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

I have v8.6.2 but haven't installed it yet.  Do you know if there are any changes to the DLLs and VM that come with v8.6.1 and v8.6.2?  I ask because if there are no differences I will install v8.6.2 and use it for all my other work.  When there are changes to our product, I just deliver a new image.  The DLLs and VM are not replaced at the user site.  If they need to be, things get more complicated.

That said I would like to move to v8.6.2 and if that makes our life easier for this LibUSB project, all the better.  If you get back to me soon on this, I will see about moving to v8.6.2.

Lou

On Friday, February 5, 2016 at 1:06:48 PM UTC-5, Seth Berman wrote:

Hi Lou,

If you have 8.6.2 image, an example of the new OSStructure capability is in MZStream (and basically everywhere else that was impacted by 64-bit).
On the class side...you see initializeAfterLoad.  However, you see that fixedSize is mysteriously not there.  Instead, you see the member names as normal, but this is followed by data type descriptors for each member name.

Notice the large number of #pointer fields.  This would be incredibly tedious to do by hand and cover 32/64 bit...and so it's not.  You just provide one definition, and then on the instance side you present the member name instead of the offset (because the offset would be different in 32/64-bit).
We technically have the information to figure out the datatype in the accessor on the instance side...but we chose not to do this because it's an extra level of indirection in what is a performance-critical area.  In the future we may provide that capability.

This framework also facilitates embedded or anonymous structures...one of the datatypes specified on #initializeOnLoad can be the name of another OSStructure....in which case it handles all offset/alignment/size computations.

There is some updated documentation for 8.6.2 in the Programmers Reference -> VA Smalltalk Virtual Machine API -> OSObject -> OSStructure section to describe this.
You can take a look here: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/stpr451.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dpr%2Fstpr451.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHRH4qRK74AB749M5LbXtb7q30pCA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dpr%2Fstpr451.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHRH4qRK74AB749M5LbXtb7q30pCA&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/stpr451.html

-- Seth

On Friday, February 5, 2016 at 12:51:29 PM UTC-5, Seth Berman wrote:

Hello Lou,

Thanks, I've imported and loaded it.

I new I would have to ask these questions...so I will now.

Do you want this to be compatible with < 8.6.2 images?

If so, then we must assume we don't have access to the new OSStructure capabilities in 8.6.2 which compute fixedSize, offsets (based on per-platform alignment rules and bitness) for you and permit the instance-size accessors to be reduced to symbolic access instead of hard-coded offsets.

The downside is

1. You would need 8.6.2 or above.

The upside is

1. You don't need separate 32/64-bit definitions or conditionals anywhere.
2. You don't need to hand compute fixedSize or any of the offsets
3. Platform-specific ABI rules are baked into the offset calculator...so you don't need to handle any adjustments to offsets based on platform-specific alignment rule

I'll leave that up to you...some of the defined structs you have are considered opaque from the header file..so you don't really need to model them as OSStructures...they can just be void * as far as your concerned.

The only structures I see that are impacted by 64-bit would be the ones that have pointers in them.  This changes the offsets for any field after that.  The platform alignment rules should probably be ok...these are typically

for double datatypes or the first structure field on AIX...and I don't see, at first glance, any impact in that area.

Let me know how you would like to proceed and I'll try to make any adjustments accordingly.

-- Seth

On Friday, February 5, 2016 at 12:25:42 PM UTC-5, Louis LaBrunda wrote:

Hey Seth,

That's great!!  Attached is version 0.01 of what I have so far.  Kernel is the only prerequisite.  I would like to end up with a package that works with both Windows and Linux.

You will need to get a copy of the LibUSB dll (I'm using libusb-1.0.dll that I think I got here: <a href="http://sourceforge.net/projects/libusb/files/libusb-1.0/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;">project libusb).  In the Pragma I name the library LibUSB.  I put the dll in the folder with my test image and point to it with: LibUSB=libusb-1.0.dll in the [PlatformLibrary Name Mappings] section if the Ini file.

If you execute "KscLibUsbApp getUsbDeviceDescriptors inspect" if it works you will get an array of device descriptors.  Lately I have been having problems with libusb_init.  I will keep playing and send another post when I figure things out or give up and ask you a question (if I can think of the right one).  For now I want to get this posted. 

All of the classes start with "KscLibUsb", I'm open to changing that by maybe removing the "Ksc" part if you like.

You are of course free to help in any way you like.  I think going over all the function definitions in the Pragma would be a good place to start.  I used the definitions found here: <a href="http://libusb.sourceforge.net/api-1.0/api.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;">functions, you can use those or any you like.

Do you know if the 64 bit version keeps the function names or add something like "64" to the end?  I ask because I expect the Pragma definitions would be different, with different names we could just put both in the same Pragma, if the names are the same we will need to think of something else, like separate packages.

If you want to make changes yourself, we will have to come up with some way to merge things.  I'm open to suggestions.

Lou

On Thursday, February 4, 2016 at 4:24:43 PM UTC-5, Seth Berman wrote:

Hi Lou,

I'd be happy to help...send it along.

-- Seth

On Thursday, February 4, 2016 at 4:13:28 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

Thanks for the offer, I will take you up on it.  I have about 100 functions defined in a Pragma and about 20 structures defined.  I have tested and have working LibUsbErrorName, LibUsbGetDeviceList, LibUsbGetDeviceDescriptor, LibUsbStrError, LibUsbGetVersion and some smalltalk methods that make calling the LibUsb functions easier.

I think most of the functions defined in the Pragma are correct.  I'm not sure about those that take structures as input or that fill in structures.  I'm less confident about the definition of the structures.  Especially those that have structures in the structures.  I also find different documentation for some structures, some are longer than others.  I'm not sure how to handle that.

I'm more than willing to do the work but it would be a big help if someone could check what I have so far.  And give me some help where my knowledge of C falls short.

Lou

Hi Seth,

I currently don't have anything deployed that uses OpenSSL, so I don't think my deployment will be impacted by moving to v8.6.2.

I do have one program that uses a fairly old DLL (RSC/Piccolo used to communicate with Tandem/HP NonStop computers).  My interface to this DLL is also very old and was built using the C External Interface parts (AbtCExternalFunction) that are dropped on the white space of a part and the function information entered into the dropped part.  From time to time I have to make changes to this program but not to the parts that deal with the DLL.  This program doesn't need to be 64 bit.  Most, if not all of my programs don't need to be 64 bit but it would probably be good to go in that direction.  How hard would it be to develop for both 32 and 64 bit?  Meaning both 32 and 64 bit versions of the same program and 32 bit versions of one program and 32/64 bit version of another.

Anyway, back to the question of moving to v8.6.2, do I have to version all open apps before I upgrade to v8.6.2?  I have separate images for each app.  Some have open apps with small changes.

Lou

"Meaning both 32 and 64 bit versions of the same program and 32 bit versions of one program and 32/64 bit version of another."

I don't follow...can you clarify?

Are you asking the difference between developing 32/64-bit compatible code using the old way vs the new way?

In any case I would say the following

1. If you have an existing 32-bit binding developed and you don't care about using it on 64-bit....then the solution is simple....do nothing.
   - We are not removing the old OSObject API...even for future releases...so it will continue to work on 32-bit vms

2. If you have an existing 32-bit binding developed and you DO care about using it on 64-bit...then you have 2 options.

•    Option 1: Continue to use hard-coded offset based APIs.
  
•  This means you will either carry around with you 2 versions of OSObjects for ones that are impacted by 64-bit or attempt to have one with a bunch of is64BitPlatform conditional checks in it...and even then you need an image that understands System is64BitPlatform (though I guess you could technically query if #is64BitPlatform is available and assume 32-bit if not....though I'll foreshadow the rest of this....we are going down a very bad path...but I'll continue)
  
• You must then figure out how to load / reference the right definition based on which vm you launched the image on.  If you want to save a 32-bit image and start it on a 64-bit image...this means you needs something dynamic....config rules won't cut it.
• You will need to hand compute any changes that impact fixed size and offsets such as field datatype (i.e. pointers), field alignment, structure alignment, packing specification, differences in struct/union definitions between 32/64-bit, and platform dependent ABI rules (i.e. default 8-byte alignment of doubles on Linux vs 4 on Windows).
• This gets more complicated when nested structures or unions are involved.
• If this is the way you want to go, I would not even attempt to figure this out by hand.  I would build a small C program that interfaces with the DLL in question and start asking sizeof and alignmentof questions in both 32/64-bit compiled version so you make sure you got the right answer to the fixedSize and member offsets.
• FYI, I did this for about 5 of the 100s (maybe 1000s) of classes I had to update for the image before I realized I wasn't going to do this....ever.

• Options 2: Use the new API which auto-calculates offset and fixed size for each platform and auto-configures on startup based on the OS and bitness of the vm it was loaded on.
• For structures, this means changing #initializeOnLoad by removing the fixedSize hard coded value (since it does it for you) and using the extended OSStructure #members:types: api to assign the members (which will already be done) and types that mirror the C code field datatypes.
• On the instance side, remove the hard-coded offset number and replace it with the symbolic field name <Symbol> instead.  So instead of 'self uint8At: 24'...it would become 'self uint8At: #fieldName'
• Unions are now first class objects with their own offset/size calculator baked in
• Nested structures are easy...just add the nested OSStructure subclass name to the types portion of the #members:types: API (examples in online doc)
• Anonymous Structures/Unions don't need to be explicitly modeled...they can be defined within the #members:types: definition (examples in online doc)
• In 8.6.2 there are a ton of examples of this.  The 64-bit impacted windows API has been re-wrapped using this and it had tons of obscure differences between 32/64 bit definitions that would have made it a nightmare to port.  In fact, I'm sure I'll still be doing it today:)
• I would look at the documentation too for OSStructures...it covers most all the use cases.  If not...I would be happy to update it

On Friday, February 5, 2016 at 3:02:28 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

I currently don't have anything deployed that uses OpenSSL, so I don't think my deployment will be impacted by moving to v8.6.2.

I do have one program that uses a fairly old DLL (RSC/Piccolo used to communicate with Tandem/HP NonStop computers).  My interface to this DLL is also very old and was built using the C External Interface parts (AbtCExternalFunction) that are dropped on the white space of a part and the function information entered into the dropped part.  From time to time I have to make changes to this program but not to the parts that deal with the DLL.  This program doesn't need to be 64 bit.  Most, if not all of my programs don't need to be 64 bit but it would probably be good to go in that direction.  How hard would it be to develop for both 32 and 64 bit?  Meaning both 32 and 64 bit versions of the same program and 32 bit versions of one program and 32/64 bit version of another.

Anyway, back to the question of moving to v8.6.2, do I have to version all open apps before I upgrade to v8.6.2?  I have separate images for each app.  Some have open apps with small changes.

Lou

On Friday, February 5, 2016 at 2:12:14 PM UTC-5, Seth Berman wrote:

Hello Lou,

There are some changes to DLLs in 8.6.2...the ones that would impact deployment are the changes to OpenSSL.

Those are referenced here in the migration guide: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8617.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8617.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEAx1F9J7kGizdjhLihB6owhzuqQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8617.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEAx1F9J7kGizdjhLihB6owhzuqQw&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8617.html

There are new envy library primitives for 8.6.2 for development images.  It probably doesn't impact your deployment, but for your development use with the ENVY repository...it requires some understanding.
Those are referenced here in the migration guide: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8612.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8612.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHN5RjN4Fv0mhkEkQ-NNTjoZmKkkg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8612.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHN5RjN4Fv0mhkEkQ-NNTjoZmKkkg&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8612.html

In short, the code you develop using the new library primitives (shipped with >= 8.6.2) will NOT be properly readable by images using the older library primitives ( < 8.6.2).
If your running a mixed environment, then the new library primitives must be copied to older versions on the product to use so they can properly decompress the >= 8.6.2 source code.

The importance of keeping as current as possible comes with your desire to use the 64-bit vm when it is released.  The 64-bit vm will definitely NOT run images older than 8.6.2 and never will.

Any third-party native library code not developed by us, and that would be impacted by 64-bit, would also need to be updated.  This is unavoidable, as the 32 to 64-bit image translator can not possibly have

knowledge of how some arbitrary 3rd party DLL is impacted by 64-bit and magically update the OSObjects.  This is why it is preferential to switch to the new OSObject capabilities sooner, rather than later, if you

have interest in running the 64-bit vm.  It reconfigures itself on startup based on the bitness so you only have 1 definition of an OSObject to support both 32 and 64-bit.

There is no doubt that the new capability makes life easier in OSObject development because hand computing anything no longer necessary....but as I said...the unavoidable cost is version compatibility.

The old way of doing things is still supported in 8.6.2 and beyond...if you stick to this it just means that things will have the potential to get messy when you try to port it to 64-bit when you must hand-compute everything and provide different definitions.

Some native libraries may have no issue (though very rare).  If some theoretical library had the same exact definitions/offsets/alignments/packing.....in both 32/64-bit for everything....then nothing would need to change.  However, I wouldn't count on this being the case:)

On Friday, February 5, 2016 at 1:19:36 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

I have v8.6.2 but haven't installed it yet.  Do you know if there are any changes to the DLLs and VM that come with v8.6.1 and v8.6.2?  I ask because if there are no differences I will install v8.6.2 and use it for all my other work.  When there are changes to our product, I just deliver a new image.  The DLLs and VM are not replaced at the user site.  If they need to be, things get more complicated.

That said I would like to move to v8.6.2 and if that makes our life easier for this LibUSB project, all the better.  If you get back to me soon on this, I will see about moving to v8.6.2.

Lou

On Friday, February 5, 2016 at 1:06:48 PM UTC-5, Seth Berman wrote:

Hi Lou,

If you have 8.6.2 image, an example of the new OSStructure capability is in MZStream (and basically everywhere else that was impacted by 64-bit).
On the class side...you see initializeAfterLoad.  However, you see that fixedSize is mysteriously not there.  Instead, you see the member names as normal, but this is followed by data type descriptors for each member name.

Notice the large number of #pointer fields.  This would be incredibly tedious to do by hand and cover 32/64 bit...and so it's not.  You just provide one definition, and then on the instance side you present the member name instead of the offset (because the offset would be different in 32/64-bit).
We technically have the information to figure out the datatype in the accessor on the instance side...but we chose not to do this because it's an extra level of indirection in what is a performance-critical area.  In the future we may provide that capability.

This framework also facilitates embedded or anonymous structures...one of the datatypes specified on #initializeOnLoad can be the name of another OSStructure....in which case it handles all offset/alignment/size computations.

There is some updated documentation for 8.6.2 in the Programmers Reference -> VA Smalltalk Virtual Machine API -> OSObject -> OSStructure section to describe this.
You can take a look here: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/stpr451.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dpr%2Fstpr451.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHRH4qRK74AB749M5LbXtb7q30pCA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dpr%2Fstpr451.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHRH4qRK74AB749M5LbXtb7q30pCA&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/stpr451.html

-- Seth

On Friday, February 5, 2016 at 12:51:29 PM UTC-5, Seth Berman wrote:

Hello Lou,

Thanks, I've imported and loaded it.

I new I would have to ask these questions...so I will now.

Do you want this to be compatible with < 8.6.2 images?

If so, then we must assume we don't have access to the new OSStructure capabilities in 8.6.2 which compute fixedSize, offsets (based on per-platform alignment rules and bitness) for you and permit the instance-size accessors to be reduced to symbolic access instead of hard-coded offsets.

The downside is

1. You would need 8.6.2 or above.

The upside is

1. You don't need separate 32/64-bit definitions or conditionals anywhere.
2. You don't need to hand compute fixedSize or any of the offsets
3. Platform-specific ABI rules are baked into the offset calculator...so you don't need to handle any adjustments to offsets based on platform-specific alignment rule

I'll leave that up to you...some of the defined structs you have are considered opaque from the header file..so you don't really need to model them as OSStructures...they can just be void * as far as your concerned.

The only structures I see that are impacted by 64-bit would be the ones that have pointers in them.  This changes the offsets for any field after that.  The platform alignment rules should probably be ok...these are typically

for double datatypes or the first structure field on AIX...and I don't see, at first glance, any impact in that area.

Let me know how you would like to proceed and I'll try to make any adjustments accordingly.

-- Seth

On Friday, February 5, 2016 at 12:25:42 PM UTC-5, Louis LaBrunda wrote:

Hey Seth,

That's great!!  Attached is version 0.01 of what I have so far.  Kernel is the only prerequisite.  I would like to end up with a package that works with both Windows and Linux.

You will need to get a copy of the LibUSB dll (I'm using libusb-1.0.dll that I think I got here: <a href="http://sourceforge.net/projects/libusb/files/libusb-1.0/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;">project libusb).  In the Pragma I name the library LibUSB.  I put the dll in the folder with my test image and point to it with: LibUSB=libusb-1.0.dll in the [PlatformLibrary Name Mappings] section if the Ini file.

If you execute "KscLibUsbApp getUsbDeviceDescriptors inspect" if it works you will get an array of device descriptors.  Lately I have been having problems with libusb_init.  I will keep playing and send another post when I figure things out or give up and ask you a question (if I can think of the right one).  For now I want to get this posted. 

All of the classes start with "KscLibUsb", I'm open to changing that by maybe removing the "Ksc" part if you like.

You are of course free to help in any way you like.  I think going over all the function definitions in the Pragma would be a good place to start.  I used the definitions found here: <a href="http://libusb.sourceforge.net/api-1.0/api.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;">functions, you can use those or any you like.

Do you know if the 64 bit version keeps the function names or add something like "64" to the end?  I ask because I expect the Pragma definitions would be different, with different names we could just put both in the same Pragma, if the names are the same we will need to think of something else, like separate packages.

If you want to make changes yourself, we will have to come up with some way to merge things.  I'm open to suggestions.

Lou

On Thursday, February 4, 2016 at 4:24:43 PM UTC-5, Seth Berman wrote:

Hi Lou,

I'd be happy to help...send it along.

-- Seth

On Thursday, February 4, 2016 at 4:13:28 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

Thanks for the offer, I will take you up on it.  I have about 100 functions defined in a Pragma and about 20 structures defined.  I have tested and have working LibUsbErrorName, LibUsbGetDeviceList, LibUsbGetDeviceDescriptor, LibUsbStrError, LibUsbGetVersion and some smalltalk methods that make calling the LibUsb functions easier.

I think most of the functions defined in the Pragma are correct.  I'm not sure about those that take structures as input or that fill in structures.  I'm less confident about the definition of the structures.  Especially those that have structures in the structures.  I also find different documentation for some structures, some are longer than others.  I'm not sure how to handle that.

I'm more than willing to do the work but it would be a big help if someone could check what I have so far.  And give me some help where my knowledge of C falls short.

Lou

Hey Seth,

Sorry about that unclear question.  I was just wondering what I had to do to develop both 32 bit and 64 bit programs once the 64 bit version comes out.  Do I need to keep both 32 bit and 64 bit development environments around?  Anyway, this doesn't effect what we are about to work on.  It is getting late now, so starting Monday I will move to v8.6.2 and we can go from there.

That would be great!!!  Will I need to do much to the Pragma?  Not that I mind.  Is there a way that the functions defined in the Pragma will match up with what the DLL functions expect?  If there is, that would also great as I won't have to guess what type/size parameters are.

Lou

P.S.  There is a second Pragma with a lot of constants defined in it.  Some of them define the sizes of things, from what you tell me, I think some of that can go away.

P.P.S.  I'm having trouble with "libusb_init" if I pass it a null pointer it works fine as it defines a default context.  If I try to pass it a pointer that it can answer/fill-in a pointer/context value I somehow mess up and can't use the pointer with libusb_exit.  I don't know how important it is to support multiple contexts but I think it would be nice to do so and I'm probably just using the wrong pointer class or using it in the wrong way.

Ok, I see.  So there will always be a 32-bit and a 64-bit image.  The issues are almost innumerable...but lets take something simple.  Objects.  In the 32-bit image a (normal) object will have a 12-byte header followed by 4 byte slots.  In a 64-bit image, this will be a 16-byte header followed by 8 byte slots.  So you can't have a single image, which consists of objects (and other things of course) that could load on both with no modification.  The good news is that we have an on-the-fly image translator that, on the 64-bit vm, notices that the image is 32-bit and converts everything to it's 64-bit format.  Then you save the image and your done...you now have a 64-bit image snapshot of your 32-bit image.  It does not go both ways...you can't load a 64-bit image on a 32-bit vm.  So I imagine the flow will be to develop in your 32-bit image...and when your ready...save the image and load in on the 64-bit vm...and then save it.  This is useful for packaged images.  You could also just load the same code from ENVY in both 32 and 64-bit images and that would work too.

I'll review the pragmas...from what I saw the formats look fine.  I have a similar layout in the new OpenSSL additions.

For libusb_init...this is one of those apis that wants the address of the pointer...or a pointer to a pointer which it will fill in.

In c you typically see this as 'libusb_context *ptr;   libusb_init(&ptr).  This &ptr is a libusb_context ** ...but is stack allocated pointer to what will be a heap allocated pointer libusb will create. We can do the same but will heap allocate the & part.

I haven't tried this yet since I have to get the libusb library...but give this a shot.

On Friday, February 5, 2016 at 4:55:16 PM UTC-5, Louis LaBrunda wrote:

Hey Seth,

Sorry about that unclear question.  I was just wondering what I had to do to develop both 32 bit and 64 bit programs once the 64 bit version comes out.  Do I need to keep both 32 bit and 64 bit development environments around?  Anyway, this doesn't effect what we are about to work on.  It is getting late now, so starting Monday I will move to v8.6.2 and we can go from there.

Compared to what I'm used to, there are not that many structures in libusb....so I'd be happy to do an 8.6.2 version of it and pass the changes back for you to see what I'm talking about.

That would be great!!!  Will I need to do much to the Pragma?  Not that I mind.  Is there a way that the functions defined in the Pragma will match up with what the DLL functions expect?  If there is, that would also great as I won't have to guess what type/size parameters are.

Lou

P.S.  There is a second Pragma with a lot of constants defined in it.  Some of them define the sizes of things, from what you tell me, I think some of that can go away.

P.P.S.  I'm having trouble with "libusb_init" if I pass it a null pointer it works fine as it defines a default context.  If I try to pass it a pointer that it can answer/fill-in a pointer/context value I somehow mess up and can't use the pointer with libusb_exit.  I don't know how important it is to support multiple contexts but I think it would be nice to do so and I'm probably just using the wrong pointer class or using it in the wrong way.

On Friday, February 5, 2016 at 4:27:13 PM UTC-5, Seth Berman wrote:

Hi Lou,

"Meaning both 32 and 64 bit versions of the same program and 32 bit versions of one program and 32/64 bit version of another."

I don't follow...can you clarify?

Are you asking the difference between developing 32/64-bit compatible code using the old way vs the new way?

In any case I would say the following

1. If you have an existing 32-bit binding developed and you don't care about using it on 64-bit....then the solution is simple....do nothing.
   - We are not removing the old OSObject API...even for future releases...so it will continue to work on 32-bit vms

2. If you have an existing 32-bit binding developed and you DO care about using it on 64-bit...then you have 2 options.

•    Option 1: Continue to use hard-coded offset based APIs.
  
•  This means you will either carry around with you 2 versions of OSObjects for ones that are impacted by 64-bit or attempt to have one with a bunch of is64BitPlatform conditional checks in it...and even then you need an image that understands System is64BitPlatform (though I guess you could technically query if #is64BitPlatform is available and assume 32-bit if not....though I'll foreshadow the rest of this....we are going down a very bad path...but I'll continue)
  
• You must then figure out how to load / reference the right definition based on which vm you launched the image on.  If you want to save a 32-bit image and start it on a 64-bit image...this means you needs something dynamic....config rules won't cut it.
• You will need to hand compute any changes that impact fixed size and offsets such as field datatype (i.e. pointers), field alignment, structure alignment, packing specification, differences in struct/union definitions between 32/64-bit, and platform dependent ABI rules (i.e. default 8-byte alignment of doubles on Linux vs 4 on Windows).
• This gets more complicated when nested structures or unions are involved.
• If this is the way you want to go, I would not even attempt to figure this out by hand.  I would build a small C program that interfaces with the DLL in question and start asking sizeof and alignmentof questions in both 32/64-bit compiled version so you make sure you got the right answer to the fixedSize and member offsets.
• FYI, I did this for about 5 of the 100s (maybe 1000s) of classes I had to update for the image before I realized I wasn't going to do this....ever.

• Options 2: Use the new API which auto-calculates offset and fixed size for each platform and auto-configures on startup based on the OS and bitness of the vm it was loaded on.
• For structures, this means changing #initializeOnLoad by removing the fixedSize hard coded value (since it does it for you) and using the extended OSStructure #members:types: api to assign the members (which will already be done) and types that mirror the C code field datatypes.
• On the instance side, remove the hard-coded offset number and replace it with the symbolic field name <Symbol> instead.  So instead of 'self uint8At: 24'...it would become 'self uint8At: #fieldName'
• Unions are now first class objects with their own offset/size calculator baked in
• Nested structures are easy...just add the nested OSStructure subclass name to the types portion of the #members:types: API (examples in online doc)
• Anonymous Structures/Unions don't need to be explicitly modeled...they can be defined within the #members:types: definition (examples in online doc)
• In 8.6.2 there are a ton of examples of this.  The 64-bit impacted windows API has been re-wrapped using this and it had tons of obscure differences between 32/64 bit definitions that would have made it a nightmare to port.  In fact, I'm sure I'll still be doing it today:)
• I would look at the documentation too for OSStructures...it covers most all the use cases.  If not...I would be happy to update it

What I'm not covering here, but equally important for 64-bit clean code, is the identification of code that just assumed pointers were 4 bytes.  I've seen everything from 'ByteArray new: 4' to store a pointer, OSUInt32 for returning or passing pointers, (temp uint32At: 0) for retrieving pointers, and even more indirect, harder to spot, ones like when windows passes in the lParam in an OSWidget callback...all the upper 32-bits, of the now 64-bits, are 1s leaving you with an incorrect value when you bitshift away the lower 16 bits to get what you thought was the upper 16 bits.

It's an unfortunate series of changes that have to be made, but that's the way things are.  The OSObject additions really help a lot, but the 'hardness' depends on both the native library and how the binding code was originally coded in Smalltalk.
Sometimes it goes really well, for example our database guy did the coding for SQLite...and I literally only had to make 1 change to make it 64-bit compatible....its nice when things work out that way.

So...maybe a little long winded...but best to get the information out there now for folks to plan for.

To your other question...no...you don't need to do anything special when loading your open-edition apps.  Unless released, any open edition classes will need to be reloaded since only the released versions of the classes will be loaded when you load the open-editition application, but that's always been the case.

Its just important to know that if you save a method in 8.6.2, and then go and look at that method in an 8.6.1 image, for example...then it will have strange characters in the 8.6.1 editor.  This means you didn't copy over your 8.6.2 library primitives to your older image installation.

Compared to what I'm used to, there are not that many structures in libusb....so I'd be happy to do an 8.6.2 version of it and pass the changes back for you to see what I'm talking about.

-- Seth

On Friday, February 5, 2016 at 3:02:28 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

I currently don't have anything deployed that uses OpenSSL, so I don't think my deployment will be impacted by moving to v8.6.2.

I do have one program that uses a fairly old DLL (RSC/Piccolo used to communicate with Tandem/HP NonStop computers).  My interface to this DLL is also very old and was built using the C External Interface parts (AbtCExternalFunction) that are dropped on the white space of a part and the function information entered into the dropped part.  From time to time I have to make changes to this program but not to the parts that deal with the DLL.  This program doesn't need to be 64 bit.  Most, if not all of my programs don't need to be 64 bit but it would probably be good to go in that direction.  How hard would it be to develop for both 32 and 64 bit?  Meaning both 32 and 64 bit versions of the same program and 32 bit versions of one program and 32/64 bit version of another.

Anyway, back to the question of moving to v8.6.2, do I have to version all open apps before I upgrade to v8.6.2?  I have separate images for each app.  Some have open apps with small changes.

Lou

On Friday, February 5, 2016 at 2:12:14 PM UTC-5, Seth Berman wrote:

Hello Lou,

There are some changes to DLLs in 8.6.2...the ones that would impact deployment are the changes to OpenSSL.

Those are referenced here in the migration guide: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8617.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8617.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEAx1F9J7kGizdjhLihB6owhzuqQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8617.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEAx1F9J7kGizdjhLihB6owhzuqQw&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8617.html

There are new envy library primitives for 8.6.2 for development images.  It probably doesn't impact your deployment, but for your development use with the ENVY repository...it requires some understanding.
Those are referenced here in the migration guide: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8612.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8612.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHN5RjN4Fv0mhkEkQ-NNTjoZmKkkg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8612.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHN5RjN4Fv0mhkEkQ-NNTjoZmKkkg&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8612.html

In short, the code you develop using the new library primitives (shipped with >= 8.6.2) will NOT be properly readable by images using the older library primitives ( < 8.6.2).
If your running a mixed environment, then the new library primitives must be copied to older versions on the product to use so they can properly decompress the >= 8.6.2 source code.

The importance of keeping as current as possible comes with your desire to use the 64-bit vm when it is released.  The 64-bit vm will definitely NOT run images older than 8.6.2 and never will.

Any third-party native library code not developed by us, and that would be impacted by 64-bit, would also need to be updated.  This is unavoidable, as the 32 to 64-bit image translator can not possibly have

knowledge of how some arbitrary 3rd party DLL is impacted by 64-bit and magically update the OSObjects.  This is why it is preferential to switch to the new OSObject capabilities sooner, rather than later, if you

have interest in running the 64-bit vm.  It reconfigures itself on startup based on the bitness so you only have 1 definition of an OSObject to support both 32 and 64-bit.

There is no doubt that the new capability makes life easier in OSObject development because hand computing anything no longer necessary....but as I said...the unavoidable cost is version compatibility.

The old way of doing things is still supported in 8.6.2 and beyond...if you stick to this it just means that things will have the potential to get messy when you try to port it to 64-bit when you must hand-compute everything and provide different definitions.

Some native libraries may have no issue (though very rare).  If some theoretical library had the same exact definitions/offsets/alignments/packing.....in both 32/64-bit for everything....then nothing would need to change.  However, I wouldn't count on this being the case:)

On Friday, February 5, 2016 at 1:19:36 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

I have v8.6.2 but haven't installed it yet.  Do you know if there are any changes to the DLLs and VM that come with v8.6.1 and v8.6.2?  I ask because if there are no differences I will install v8.6.2 and use it for all my other work.  When there are changes to our product, I just deliver a new image.  The DLLs and VM are not replaced at the user site.  If they need to be, things get more complicated.

That said I would like to move to v8.6.2 and if that makes our life easier for this LibUSB project, all the better.  If you get back to me soon on this, I will see about moving to v8.6.2.

Lou

On Friday, February 5, 2016 at 1:06:48 PM UTC-5, Seth Berman wrote:

Hi Lou,

If you have 8.6.2 image, an example of the new OSStructure capability is in MZStream (and basically everywhere else that was impacted by 64-bit).
On the class side...you see initializeAfterLoad.  However, you see that fixedSize is mysteriously not there.  Instead, you see the member names as normal, but this is followed by data type descriptors for each member name.

Notice the large number of #pointer fields.  This would be incredibly tedious to do by hand and cover 32/64 bit...and so it's not.  You just provide one definition, and then on the instance side you present the member name instead of the offset (because the offset would be different in 32/64-bit).
We technically have the information to figure out the datatype in the accessor on the instance side...but we chose not to do this because it's an extra level of indirection in what is a performance-critical area.  In the future we may provide that capability.

This framework also facilitates embedded or anonymous structures...one of the datatypes specified on #initializeOnLoad can be the name of another OSStructure....in which case it handles all offset/alignment/size computations.

There is some updated documentation for 8.6.2 in the Programmers Reference -> VA Smalltalk Virtual Machine API -> OSObject -> OSStructure section to describe this.
You can take a look here: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/stpr451.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dpr%2Fstpr451.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHRH4qRK74AB749M5LbXtb7q30pCA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dpr%2Fstpr451.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHRH4qRK74AB749M5LbXtb7q30pCA&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/stpr451.html

-- Seth

On Friday, February 5, 2016 at 12:51:29 PM UTC-5, Seth Berman wrote:

Hello Lou,

Thanks, I've imported and loaded it.

I new I would have to ask these questions...so I will now.

Do you want this to be compatible with < 8.6.2 images?

If so, then we must assume we don't have access to the new OSStructure capabilities in 8.6.2 which compute fixedSize, offsets (based on per-platform alignment rules and bitness) for you and permit the instance-size accessors to be reduced to symbolic access instead of hard-coded offsets.

The downside is

1. You would need 8.6.2 or above.

The upside is

1. You don't need separate 32/64-bit definitions or conditionals anywhere.
2. You don't need to hand compute fixedSize or any of the offsets
3. Platform-specific ABI rules are baked into the offset calculator...so you don't need to handle any adjustments to offsets based on platform-specific alignment rule

I'll leave that up to you...some of the defined structs you have are considered opaque from the header file..so you don't really need to model them as OSStructures...they can just be void * as far as your concerned.

The only structures I see that are impacted by 64-bit would be the ones that have pointers in them.  This changes the offsets for any field after that.  The platform alignment rules should probably be ok...these are typically

for double datatypes or the first structure field on AIX...and I don't see, at first glance, any impact in that area.

Let me know how you would like to proceed and I'll try to make any adjustments accordingly.

-- Seth

On Friday, February 5, 2016 at 12:25:42 PM UTC-5, Louis LaBrunda wrote:

Hey Seth,

That's great!!  Attached is version 0.01 of what I have so far.  Kernel is the only prerequisite.  I would like to end up with a package that works with both Windows and Linux.

You will need to get a copy of the LibUSB dll (I'm using libusb-1.0.dll that I think I got here: <a href="http://sourceforge.net/projects/libusb/files/libusb-1.0/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;">project libusb).  In the Pragma I name the library LibUSB.  I put the dll in the folder with my test image and point to it with: LibUSB=libusb-1.0.dll in the [PlatformLibrary Name Mappings] section if the Ini file.

If you execute "KscLibUsbApp getUsbDeviceDescriptors inspect" if it works you will get an array of device descriptors.  Lately I have been having problems with libusb_init.  I will keep playing and send another post when I figure things out or give up and ask you a question (if I can think of the right one).  For now I want to get this posted. 

All of the classes start with "KscLibUsb", I'm open to changing that by maybe removing the "Ksc" part if you like.

You are of course free to help in any way you like.  I think going over all the function definitions in the Pragma would be a good place to start.  I used the definitions found here: <a href="http://libusb.sourceforge.net/api-1.0/api.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;">functions, you can use those or any you like.

Do you know if the 64 bit version keeps the function names or add something like "64" to the end?  I ask because I expect the Pragma definitions would be different, with different names we could just put both in the same Pragma, if the names are the same we will need to think of something else, like separate packages.

If you want to make changes yourself, we will have to come up with some way to merge things.  I'm open to suggestions.

Lou

On Thursday, February 4, 2016 at 4:24:43 PM UTC-5, Seth Berman wrote:

Hi Lou,

I'd be happy to help...send it along.

-- Seth

On Thursday, February 4, 2016 at 4:13:28 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

Thanks for the offer, I will take you up on it.  I have about 100 functions defined in a Pragma and about 20 structures defined.  I have tested and have working LibUsbErrorName, LibUsbGetDeviceList, LibUsbGetDeviceDescriptor, LibUsbStrError, LibUsbGetVersion and some smalltalk methods that make calling the LibUsb functions easier.

I think most of the functions defined in the Pragma are correct.  I'm not sure about those that take structures as input or that fill in structures.  I'm less confident about the definition of the structures.  Especially those that have structures in the structures.  I also find different documentation for some structures, some are longer than others.  I'm not sure how to handle that.

I'm more than willing to do the work but it would be a big help if someone could check what I have so far.  And give me some help where my knowledge of C falls short.

Lou

Hey Seth,

Thanks for all the info on v8.6.2 and the forthcoming 64 bit VM.

After looking at your version of #usbContext, I think it is not a very good idea to lazy initialize UsbContext.  Instead I propose we have two methods #initDefaultContext and #initNewContext.  #usbContext would just return UsbContext.  #initNewContext will do much of what you have in #usbContext.  #initDefaultContext will call libusb_init with a null pointer.  Both will set UsbContext and return what ever libusb_init returns.  This way we don't have to deal with any errors, we just pass them back to the user.  Methods like my #getUsbDeviceDescriptors can check to see if there is a context and if not call #initDefaultContext.

Can you check to see if #KscLibUsbContext is sub classed from the right class.

Okay, Monday I upgrade to v8.6.2.  I look forward to your LibUSB structures.

Lou

Quick question...what version of libusb are you using to bind to?  There are certain things you have that look like this is a binding to a very old version.

I'm using the 1.0.20 version released in September 2015.

-- Seth

On Saturday, February 6, 2016 at 2:18:09 PM UTC-5, Louis LaBrunda wrote:

Hey Seth,

Thanks for all the info on v8.6.2 and the forthcoming 64 bit VM.

After looking at your version of #usbContext, I think it is not a very good idea to lazy initialize UsbContext.  Instead I propose we have two methods #initDefaultContext and #initNewContext.  #usbContext would just return UsbContext.  #initNewContext will do much of what you have in #usbContext.  #initDefaultContext will call libusb_init with a null pointer.  Both will set UsbContext and return what ever libusb_init returns.  This way we don't have to deal with any errors, we just pass them back to the user.  Methods like my #getUsbDeviceDescriptors can check to see if there is a context and if not call #initDefaultContext.

Can you check to see if #KscLibUsbContext is sub classed from the right class.

Okay, Monday I upgrade to v8.6.2.  I look forward to your LibUSB structures.

Lou

On Friday, February 5, 2016 at 6:00:46 PM UTC-5, Seth Berman wrote:

Hi Lou,

Ok, I see.  So there will always be a 32-bit and a 64-bit image.  The issues are almost innumerable...but lets take something simple.  Objects.  In the 32-bit image a (normal) object will have a 12-byte header followed by 4 byte slots.  In a 64-bit image, this will be a 16-byte header followed by 8 byte slots.  So you can't have a single image, which consists of objects (and other things of course) that could load on both with no modification.  The good news is that we have an on-the-fly image translator that, on the 64-bit vm, notices that the image is 32-bit and converts everything to it's 64-bit format.  Then you save the image and your done...you now have a 64-bit image snapshot of your 32-bit image.  It does not go both ways...you can't load a 64-bit image on a 32-bit vm.  So I imagine the flow will be to develop in your 32-bit image...and when your ready...save the image and load in on the 64-bit vm...and then save it.  This is useful for packaged images.  You could also just load the same code from ENVY in both 32 and 64-bit images and that would work too.

I'll review the pragmas...from what I saw the formats look fine.  I have a similar layout in the new OpenSSL additions.

For libusb_init...this is one of those apis that wants the address of the pointer...or a pointer to a pointer which it will fill in.

In c you typically see this as 'libusb_context *ptr;   libusb_init(&ptr).  This &ptr is a libusb_context ** ...but is stack allocated pointer to what will be a heap allocated pointer libusb will create. We can do the same but will heap allocate the & part.

I haven't tried this yet since I have to get the libusb library...but give this a shot.

usbContext

"Answer our UsbContext."

UsbContext isNil

ifTrue: [| contextPtrPtr |

contextPtrPtr := OSObjectPointer calloc: 1 itemType: KscLibUsbContext.

(LibUsbInit callWith: contextPtrPtr) = 0

ifTrue: [UsbContext := contextPtrPtr at: 0]

ifFalse: [ "Explicit Fail Condition?" ].

contextPtrPtr free "Free the pointer we created....not the one inside that is now pointed to by UsbContext"].

^UsbContext

On Friday, February 5, 2016 at 4:55:16 PM UTC-5, Louis LaBrunda wrote:

Hey Seth,

Sorry about that unclear question.  I was just wondering what I had to do to develop both 32 bit and 64 bit programs once the 64 bit version comes out.  Do I need to keep both 32 bit and 64 bit development environments around?  Anyway, this doesn't effect what we are about to work on.  It is getting late now, so starting Monday I will move to v8.6.2 and we can go from there.

Compared to what I'm used to, there are not that many structures in libusb....so I'd be happy to do an 8.6.2 version of it and pass the changes back for you to see what I'm talking about.

That would be great!!!  Will I need to do much to the Pragma?  Not that I mind.  Is there a way that the functions defined in the Pragma will match up with what the DLL functions expect?  If there is, that would also great as I won't have to guess what type/size parameters are.

Lou

P.S.  There is a second Pragma with a lot of constants defined in it.  Some of them define the sizes of things, from what you tell me, I think some of that can go away.

P.P.S.  I'm having trouble with "libusb_init" if I pass it a null pointer it works fine as it defines a default context.  If I try to pass it a pointer that it can answer/fill-in a pointer/context value I somehow mess up and can't use the pointer with libusb_exit.  I don't know how important it is to support multiple contexts but I think it would be nice to do so and I'm probably just using the wrong pointer class or using it in the wrong way.

On Friday, February 5, 2016 at 4:27:13 PM UTC-5, Seth Berman wrote:

Hi Lou,

"Meaning both 32 and 64 bit versions of the same program and 32 bit versions of one program and 32/64 bit version of another."

I don't follow...can you clarify?

Are you asking the difference between developing 32/64-bit compatible code using the old way vs the new way?

In any case I would say the following

1. If you have an existing 32-bit binding developed and you don't care about using it on 64-bit....then the solution is simple....do nothing.
   - We are not removing the old OSObject API...even for future releases...so it will continue to work on 32-bit vms

2. If you have an existing 32-bit binding developed and you DO care about using it on 64-bit...then you have 2 options.

•    Option 1: Continue to use hard-coded offset based APIs.
  
•  This means you will either carry around with you 2 versions of OSObjects for ones that are impacted by 64-bit or attempt to have one with a bunch of is64BitPlatform conditional checks in it...and even then you need an image that understands System is64BitPlatform (though I guess you could technically query if #is64BitPlatform is available and assume 32-bit if not....though I'll foreshadow the rest of this....we are going down a very bad path...but I'll continue)
  
• You must then figure out how to load / reference the right definition based on which vm you launched the image on.  If you want to save a 32-bit image and start it on a 64-bit image...this means you needs something dynamic....config rules won't cut it.
• You will need to hand compute any changes that impact fixed size and offsets such as field datatype (i.e. pointers), field alignment, structure alignment, packing specification, differences in struct/union definitions between 32/64-bit, and platform dependent ABI rules (i.e. default 8-byte alignment of doubles on Linux vs 4 on Windows).
• This gets more complicated when nested structures or unions are involved.
• If this is the way you want to go, I would not even attempt to figure this out by hand.  I would build a small C program that interfaces with the DLL in question and start asking sizeof and alignmentof questions in both 32/64-bit compiled version so you make sure you got the right answer to the fixedSize and member offsets.
• FYI, I did this for about 5 of the 100s (maybe 1000s) of classes I had to update for the image before I realized I wasn't going to do this....ever.

• Options 2: Use the new API which auto-calculates offset and fixed size for each platform and auto-configures on startup based on the OS and bitness of the vm it was loaded on.
• For structures, this means changing #initializeOnLoad by removing the fixedSize hard coded value (since it does it for you) and using the extended OSStructure #members:types: api to assign the members (which will already be done) and types that mirror the C code field datatypes.
• On the instance side, remove the hard-coded offset number and replace it with the symbolic field name <Symbol> instead.  So instead of 'self uint8At: 24'...it would become 'self uint8At: #fieldName'
• Unions are now first class objects with their own offset/size calculator baked in
• Nested structures are easy...just add the nested OSStructure subclass name to the types portion of the #members:types: API (examples in online doc)
• Anonymous Structures/Unions don't need to be explicitly modeled...they can be defined within the #members:types: definition (examples in online doc)
• In 8.6.2 there are a ton of examples of this.  The 64-bit impacted windows API has been re-wrapped using this and it had tons of obscure differences between 32/64 bit definitions that would have made it a nightmare to port.  In fact, I'm sure I'll still be doing it today:)
• I would look at the documentation too for OSStructures...it covers most all the use cases.  If not...I would be happy to update it

What I'm not covering here, but equally important for 64-bit clean code, is the identification of code that just assumed pointers were 4 bytes.  I've seen everything from 'ByteArray new: 4' to store a pointer, OSUInt32 for returning or passing pointers, (temp uint32At: 0) for retrieving pointers, and even more indirect, harder to spot, ones like when windows passes in the lParam in an OSWidget callback...all the upper 32-bits, of the now 64-bits, are 1s leaving you with an incorrect value when you bitshift away the lower 16 bits to get what you thought was the upper 16 bits.

It's an unfortunate series of changes that have to be made, but that's the way things are.  The OSObject additions really help a lot, but the 'hardness' depends on both the native library and how the binding code was originally coded in Smalltalk.
Sometimes it goes really well, for example our database guy did the coding for SQLite...and I literally only had to make 1 change to make it 64-bit compatible....its nice when things work out that way.

So...maybe a little long winded...but best to get the information out there now for folks to plan for.

To your other question...no...you don't need to do anything special when loading your open-edition apps.  Unless released, any open edition classes will need to be reloaded since only the released versions of the classes will be loaded when you load the open-editition application, but that's always been the case.

Its just important to know that if you save a method in 8.6.2, and then go and look at that method in an 8.6.1 image, for example...then it will have strange characters in the 8.6.1 editor.  This means you didn't copy over your 8.6.2 library primitives to your older image installation.

Compared to what I'm used to, there are not that many structures in libusb....so I'd be happy to do an 8.6.2 version of it and pass the changes back for you to see what I'm talking about.

-- Seth

On Friday, February 5, 2016 at 3:02:28 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

I currently don't have anything deployed that uses OpenSSL, so I don't think my deployment will be impacted by moving to v8.6.2.

I do have one program that uses a fairly old DLL (RSC/Piccolo used to communicate with Tandem/HP NonStop computers).  My interface to this DLL is also very old and was built using the C External Interface parts (AbtCExternalFunction) that are dropped on the white space of a part and the function information entered into the dropped part.  From time to time I have to make changes to this program but not to the parts that deal with the DLL.  This program doesn't need to be 64 bit.  Most, if not all of my programs don't need to be 64 bit but it would probably be good to go in that direction.  How hard would it be to develop for both 32 and 64 bit?  Meaning both 32 and 64 bit versions of the same program and 32 bit versions of one program and 32/64 bit version of another.

Anyway, back to the question of moving to v8.6.2, do I have to version all open apps before I upgrade to v8.6.2?  I have separate images for each app.  Some have open apps with small changes.

Lou

On Friday, February 5, 2016 at 2:12:14 PM UTC-5, Seth Berman wrote:

Hello Lou,

There are some changes to DLLs in 8.6.2...the ones that would impact deployment are the changes to OpenSSL.

Those are referenced here in the migration guide: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8617.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8617.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEAx1F9J7kGizdjhLihB6owhzuqQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8617.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEAx1F9J7kGizdjhLihB6owhzuqQw&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8617.html

There are new envy library primitives for 8.6.2 for development images.  It probably doesn't impact your deployment, but for your development use with the ENVY repository...it requires some understanding.
Those are referenced here in the migration guide: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8612.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8612.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHN5RjN4Fv0mhkEkQ-NNTjoZmKkkg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8612.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHN5RjN4Fv0mhkEkQ-NNTjoZmKkkg&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8612.html

In short, the code you develop using the new library primitives (shipped with >= 8.6.2) will NOT be properly readable by images using the older library primitives ( < 8.6.2).
If your running a mixed environment, then the new library primitives must be copied to older versions on the product to use so they can properly decompress the >= 8.6.2 source code.

The importance of keeping as current as possible comes with your desire to use the 64-bit vm when it is released.  The 64-bit vm will definitely NOT run images older than 8.6.2 and never will.

Any third-party native library code not developed by us, and that would be impacted by 64-bit, would also need to be updated.  This is unavoidable, as the 32 to 64-bit image translator can not possibly have

knowledge of how some arbitrary 3rd party DLL is impacted by 64-bit and magically update the OSObjects.  This is why it is preferential to switch to the new OSObject capabilities sooner, rather than later, if you

have interest in running the 64-bit vm.  It reconfigures itself on startup based on the bitness so you only have 1 definition of an OSObject to support both 32 and 64-bit.

There is no doubt that the new capability makes life easier in OSObject development because hand computing anything no longer necessary....but as I said...the unavoidable cost is version compatibility.

The old way of doing things is still supported in 8.6.2 and beyond...if you stick to this it just means that things will have the potential to get messy when you try to port it to 64-bit when you must hand-compute everything and provide different definitions.

Some native libraries may have no issue (though very rare).  If some theoretical library had the same exact definitions/offsets/alignments/packing.....in both 32/64-bit for everything....then nothing would need to change.  However, I wouldn't count on this being the case:)

On Friday, February 5, 2016 at 1:19:36 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

I have v8.6.2 but haven't installed it yet.  Do you know if there are any changes to the DLLs and VM that come with v8.6.1 and v8.6.2?  I ask because if there are no differences I will install v8.6.2 and use it for all my other work.  When there are changes to our product, I just deliver a new image.  The DLLs and VM are not replaced at the user site.  If they need to be, things get more complicated.

That said I would like to move to v8.6.2 and if that makes our life easier for this LibUSB project, all the better.  If you get back to me soon on this, I will see about moving to v8.6.2.

Lou

On Friday, February 5, 2016 at 1:06:48 PM UTC-5, Seth Berman wrote:

Hi Lou,

If you have 8.6.2 image, an example of the new OSStructure capability is in MZStream (and basically everywhere else that was impacted by 64-bit).
On the class side...you see initializeAfterLoad.  However, you see that fixedSize is mysteriously not there.  Instead, you see the member names as normal, but this is followed by data type descriptors for each member name.

Notice the large number of #pointer fields.  This would be incredibly tedious to do by hand and cover 32/64 bit...and so it's not.  You just provide one definition, and then on the instance side you present the member name instead of the offset (because the offset would be different in 32/64-bit).
We technically have the information to figure out the datatype in the accessor on the instance side...but we chose not to do this because it's an extra level of indirection in what is a performance-critical area.  In the future we may provide that capability.

This framework also facilitates embedded or anonymous structures...one of the datatypes specified on #initializeOnLoad can be the name of another OSStructure....in which case it handles all offset/alignment/size computations.

There is some updated documentation for 8.6.2 in the Programmers Reference -> VA Smalltalk Virtual Machine API -> OSObject -> OSStructure section to describe this.
You can take a look here: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/stpr451.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dpr%2Fstpr451.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHRH4qRK74AB749M5LbXtb7q30pCA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dpr%2Fstpr451.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHRH4qRK74AB749M5LbXtb7q30pCA&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/stpr451.html

-- Seth

On Friday, February 5, 2016 at 12:51:29 PM UTC-5, Seth Berman wrote:

Hello Lou,

Thanks, I've imported and loaded it.

I new I would have to ask these questions...so I will now.

Do you want this to be compatible with < 8.6.2 images?

If so, then we must assume we don't have access to the new OSStructure capabilities in 8.6.2 which compute fixedSize, offsets (based on per-platform alignment rules and bitness) for you and permit the instance-size accessors to be reduced to symbolic access instead of hard-coded offsets.

The downside is

1. You would need 8.6.2 or above.

The upside is

1. You don't need separate 32/64-bit definitions or conditionals anywhere.
2. You don't need to hand compute fixedSize or any of the offsets
3. Platform-specific ABI rules are baked into the offset calculator...so you don't need to handle any adjustments to offsets based on platform-specific alignment rule

I'll leave that up to you...some of the defined structs you have are considered opaque from the header file..so you don't really need to model them as OSStructures...they can just be void * as far as your concerned.

The only structures I see that are impacted by 64-bit would be the ones that have pointers in them.  This changes the offsets for any field after that.  The platform alignment rules should probably be ok...these are typically

for double datatypes or the first structure field on AIX...and I don't see, at first glance, any impact in that area.

Let me know how you would like to proceed and I'll try to make any adjustments accordingly.

-- Seth

On Friday, February 5, 2016 at 12:25:42 PM UTC-5, Louis LaBrunda wrote:

Hey Seth,

That's great!!  Attached is version 0.01 of what I have so far.  Kernel is the only prerequisite.  I would like to end up with a package that works with both Windows and Linux.

You will need to get a copy of the LibUSB dll (I'm using libusb-1.0.dll that I think I got here: <a href="http://sourceforge.net/projects/libusb/files/libusb-1.0/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;">project libusb).  In the Pragma I name the library LibUSB.  I put the dll in the folder with my test image and point to it with: LibUSB=libusb-1.0.dll in the [PlatformLibrary Name Mappings] section if the Ini file.

If you execute "KscLibUsbApp getUsbDeviceDescriptors inspect" if it works you will get an array of device descriptors.  Lately I have been having problems with libusb_init.  I will keep playing and send another post when I figure things out or give up and ask you a question (if I can think of the right one).  For now I want to get this posted. 

All of the classes start with "KscLibUsb", I'm open to changing that by maybe removing the "Ksc" part if you like.

You are of course free to help in any way you like.  I think going over all the function definitions in the Pragma would be a good place to start.  I used the definitions found here: <a href="http://libusb.sourceforge.net/api-1.0/api.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;">functions, you can use those or any you like.

Do you know if the 64 bit version keeps the function names or add something like "64" to the end?  I ask because I expect the Pragma definitions would be different, with different names we could just put both in the same Pragma, if the names are the same we will need to think of something else, like separate packages.

If you want to make changes yourself, we will have to come up with some way to merge things.  I'm open to suggestions.

Lou

On Thursday, February 4, 2016 at 4:24:43 PM UTC-5, Seth Berman wrote:

Hi Lou,

I'd be happy to help...send it along.

-- Seth

On Thursday, February 4, 2016 at 4:13:28 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

Thanks for the offer, I will take you up on it.  I have about 100 functions defined in a Pragma and about 20 structures defined.  I have tested and have working LibUsbErrorName, LibUsbGetDeviceList, LibUsbGetDeviceDescriptor, LibUsbStrError, LibUsbGetVersion and some smalltalk methods that make calling the LibUsb functions easier.

I think most of the functions defined in the Pragma are correct.  I'm not sure about those that take structures as input or that fill in structures.  I'm less confident about the definition of the structures.  Especially those that have structures in the structures.  I also find different documentation for some structures, some are longer than others.  I'm not sure how to handle that.

I'm more than willing to do the work but it would be a big help if someone could check what I have so far.  And give me some help where my knowledge of C falls short.

Lou