Bug in OSPtr #address and friends or is it just my lack of C knowledge?

Hey Seth,

Thanks for all the info on v8.6.2 and the forthcoming 64 bit VM.

After looking at your version of #usbContext, I think it is not a very good idea to lazy initialize UsbContext.  Instead I propose we have two methods #initDefaultContext and #initNewContext.  #usbContext would just return UsbContext.  #initNewContext will do much of what you have in #usbContext.  #initDefaultContext will call libusb_init with a null pointer.  Both will set UsbContext and return what ever libusb_init returns.  This way we don't have to deal with any errors, we just pass them back to the user.  Methods like my #getUsbDeviceDescriptors can check to see if there is a context and if not call #initDefaultContext.

Can you check to see if #KscLibUsbContext is sub classed from the right class.

Okay, Monday I upgrade to v8.6.2.  I look forward to your LibUSB structures.

Lou

On Friday, February 5, 2016 at 6:00:46 PM UTC-5, Seth Berman wrote:

Hi Lou,

Ok, I see.  So there will always be a 32-bit and a 64-bit image.  The issues are almost innumerable...but lets take something simple.  Objects.  In the 32-bit image a (normal) object will have a 12-byte header followed by 4 byte slots.  In a 64-bit image, this will be a 16-byte header followed by 8 byte slots.  So you can't have a single image, which consists of objects (and other things of course) that could load on both with no modification.  The good news is that we have an on-the-fly image translator that, on the 64-bit vm, notices that the image is 32-bit and converts everything to it's 64-bit format.  Then you save the image and your done...you now have a 64-bit image snapshot of your 32-bit image.  It does not go both ways...you can't load a 64-bit image on a 32-bit vm.  So I imagine the flow will be to develop in your 32-bit image...and when your ready...save the image and load in on the 64-bit vm...and then save it.  This is useful for packaged images.  You could also just load the same code from ENVY in both 32 and 64-bit images and that would work too.

I'll review the pragmas...from what I saw the formats look fine.  I have a similar layout in the new OpenSSL additions.

For libusb_init...this is one of those apis that wants the address of the pointer...or a pointer to a pointer which it will fill in.

In c you typically see this as 'libusb_context *ptr;   libusb_init(&ptr).  This &ptr is a libusb_context ** ...but is stack allocated pointer to what will be a heap allocated pointer libusb will create. We can do the same but will heap allocate the & part.

I haven't tried this yet since I have to get the libusb library...but give this a shot.

usbContext

"Answer our UsbContext."

UsbContext isNil

ifTrue: [| contextPtrPtr |

contextPtrPtr := OSObjectPointer calloc: 1 itemType: KscLibUsbContext.

(LibUsbInit callWith: contextPtrPtr) = 0

ifTrue: [UsbContext := contextPtrPtr at: 0]

ifFalse: [ "Explicit Fail Condition?" ].

contextPtrPtr free "Free the pointer we created....not the one inside that is now pointed to by UsbContext"].

^UsbContext

On Friday, February 5, 2016 at 4:55:16 PM UTC-5, Louis LaBrunda wrote:

Hey Seth,

Sorry about that unclear question.  I was just wondering what I had to do to develop both 32 bit and 64 bit programs once the 64 bit version comes out.  Do I need to keep both 32 bit and 64 bit development environments around?  Anyway, this doesn't effect what we are about to work on.  It is getting late now, so starting Monday I will move to v8.6.2 and we can go from there.

Compared to what I'm used to, there are not that many structures in libusb....so I'd be happy to do an 8.6.2 version of it and pass the changes back for you to see what I'm talking about.

That would be great!!!  Will I need to do much to the Pragma?  Not that I mind.  Is there a way that the functions defined in the Pragma will match up with what the DLL functions expect?  If there is, that would also great as I won't have to guess what type/size parameters are.

Lou

P.S.  There is a second Pragma with a lot of constants defined in it.  Some of them define the sizes of things, from what you tell me, I think some of that can go away.

P.P.S.  I'm having trouble with "libusb_init" if I pass it a null pointer it works fine as it defines a default context.  If I try to pass it a pointer that it can answer/fill-in a pointer/context value I somehow mess up and can't use the pointer with libusb_exit.  I don't know how important it is to support multiple contexts but I think it would be nice to do so and I'm probably just using the wrong pointer class or using it in the wrong way.

On Friday, February 5, 2016 at 4:27:13 PM UTC-5, Seth Berman wrote:

Hi Lou,

"Meaning both 32 and 64 bit versions of the same program and 32 bit versions of one program and 32/64 bit version of another."

I don't follow...can you clarify?

Are you asking the difference between developing 32/64-bit compatible code using the old way vs the new way?

In any case I would say the following

1. If you have an existing 32-bit binding developed and you don't care about using it on 64-bit....then the solution is simple....do nothing.
   - We are not removing the old OSObject API...even for future releases...so it will continue to work on 32-bit vms

2. If you have an existing 32-bit binding developed and you DO care about using it on 64-bit...then you have 2 options.

•    Option 1: Continue to use hard-coded offset based APIs.
  
•  This means you will either carry around with you 2 versions of OSObjects for ones that are impacted by 64-bit or attempt to have one with a bunch of is64BitPlatform conditional checks in it...and even then you need an image that understands System is64BitPlatform (though I guess you could technically query if #is64BitPlatform is available and assume 32-bit if not....though I'll foreshadow the rest of this....we are going down a very bad path...but I'll continue)
  
• You must then figure out how to load / reference the right definition based on which vm you launched the image on.  If you want to save a 32-bit image and start it on a 64-bit image...this means you needs something dynamic....config rules won't cut it.
• You will need to hand compute any changes that impact fixed size and offsets such as field datatype (i.e. pointers), field alignment, structure alignment, packing specification, differences in struct/union definitions between 32/64-bit, and platform dependent ABI rules (i.e. default 8-byte alignment of doubles on Linux vs 4 on Windows).
• This gets more complicated when nested structures or unions are involved.
• If this is the way you want to go, I would not even attempt to figure this out by hand.  I would build a small C program that interfaces with the DLL in question and start asking sizeof and alignmentof questions in both 32/64-bit compiled version so you make sure you got the right answer to the fixedSize and member offsets.
• FYI, I did this for about 5 of the 100s (maybe 1000s) of classes I had to update for the image before I realized I wasn't going to do this....ever.

• Options 2: Use the new API which auto-calculates offset and fixed size for each platform and auto-configures on startup based on the OS and bitness of the vm it was loaded on.
• For structures, this means changing #initializeOnLoad by removing the fixedSize hard coded value (since it does it for you) and using the extended OSStructure #members:types: api to assign the members (which will already be done) and types that mirror the C code field datatypes.
• On the instance side, remove the hard-coded offset number and replace it with the symbolic field name <Symbol> instead.  So instead of 'self uint8At: 24'...it would become 'self uint8At: #fieldName'
• Unions are now first class objects with their own offset/size calculator baked in
• Nested structures are easy...just add the nested OSStructure subclass name to the types portion of the #members:types: API (examples in online doc)
• Anonymous Structures/Unions don't need to be explicitly modeled...they can be defined within the #members:types: definition (examples in online doc)
• In 8.6.2 there are a ton of examples of this.  The 64-bit impacted windows API has been re-wrapped using this and it had tons of obscure differences between 32/64 bit definitions that would have made it a nightmare to port.  In fact, I'm sure I'll still be doing it today:)
• I would look at the documentation too for OSStructures...it covers most all the use cases.  If not...I would be happy to update it

What I'm not covering here, but equally important for 64-bit clean code, is the identification of code that just assumed pointers were 4 bytes.  I've seen everything from 'ByteArray new: 4' to store a pointer, OSUInt32 for returning or passing pointers, (temp uint32At: 0) for retrieving pointers, and even more indirect, harder to spot, ones like when windows passes in the lParam in an OSWidget callback...all the upper 32-bits, of the now 64-bits, are 1s leaving you with an incorrect value when you bitshift away the lower 16 bits to get what you thought was the upper 16 bits.

It's an unfortunate series of changes that have to be made, but that's the way things are.  The OSObject additions really help a lot, but the 'hardness' depends on both the native library and how the binding code was originally coded in Smalltalk.
Sometimes it goes really well, for example our database guy did the coding for SQLite...and I literally only had to make 1 change to make it 64-bit compatible....its nice when things work out that way.

So...maybe a little long winded...but best to get the information out there now for folks to plan for.

To your other question...no...you don't need to do anything special when loading your open-edition apps.  Unless released, any open edition classes will need to be reloaded since only the released versions of the classes will be loaded when you load the open-editition application, but that's always been the case.

Its just important to know that if you save a method in 8.6.2, and then go and look at that method in an 8.6.1 image, for example...then it will have strange characters in the 8.6.1 editor.  This means you didn't copy over your 8.6.2 library primitives to your older image installation.

Compared to what I'm used to, there are not that many structures in libusb....so I'd be happy to do an 8.6.2 version of it and pass the changes back for you to see what I'm talking about.

-- Seth

On Friday, February 5, 2016 at 3:02:28 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

I currently don't have anything deployed that uses OpenSSL, so I don't think my deployment will be impacted by moving to v8.6.2.

I do have one program that uses a fairly old DLL (RSC/Piccolo used to communicate with Tandem/HP NonStop computers).  My interface to this DLL is also very old and was built using the C External Interface parts (AbtCExternalFunction) that are dropped on the white space of a part and the function information entered into the dropped part.  From time to time I have to make changes to this program but not to the parts that deal with the DLL.  This program doesn't need to be 64 bit.  Most, if not all of my programs don't need to be 64 bit but it would probably be good to go in that direction.  How hard would it be to develop for both 32 and 64 bit?  Meaning both 32 and 64 bit versions of the same program and 32 bit versions of one program and 32/64 bit version of another.

Anyway, back to the question of moving to v8.6.2, do I have to version all open apps before I upgrade to v8.6.2?  I have separate images for each app.  Some have open apps with small changes.

Lou

On Friday, February 5, 2016 at 2:12:14 PM UTC-5, Seth Berman wrote:

Hello Lou,

There are some changes to DLLs in 8.6.2...the ones that would impact deployment are the changes to OpenSSL.

Those are referenced here in the migration guide: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8617.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8617.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEAx1F9J7kGizdjhLihB6owhzuqQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8617.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEAx1F9J7kGizdjhLihB6owhzuqQw&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8617.html

There are new envy library primitives for 8.6.2 for development images.  It probably doesn't impact your deployment, but for your development use with the ENVY repository...it requires some understanding.
Those are referenced here in the migration guide: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8612.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8612.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHN5RjN4Fv0mhkEkQ-NNTjoZmKkkg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8612.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHN5RjN4Fv0mhkEkQ-NNTjoZmKkkg&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8612.html

In short, the code you develop using the new library primitives (shipped with >= 8.6.2) will NOT be properly readable by images using the older library primitives ( < 8.6.2).
If your running a mixed environment, then the new library primitives must be copied to older versions on the product to use so they can properly decompress the >= 8.6.2 source code.

The importance of keeping as current as possible comes with your desire to use the 64-bit vm when it is released.  The 64-bit vm will definitely NOT run images older than 8.6.2 and never will.

Any third-party native library code not developed by us, and that would be impacted by 64-bit, would also need to be updated.  This is unavoidable, as the 32 to 64-bit image translator can not possibly have

knowledge of how some arbitrary 3rd party DLL is impacted by 64-bit and magically update the OSObjects.  This is why it is preferential to switch to the new OSObject capabilities sooner, rather than later, if you

have interest in running the 64-bit vm.  It reconfigures itself on startup based on the bitness so you only have 1 definition of an OSObject to support both 32 and 64-bit.

There is no doubt that the new capability makes life easier in OSObject development because hand computing anything no longer necessary....but as I said...the unavoidable cost is version compatibility.

The old way of doing things is still supported in 8.6.2 and beyond...if you stick to this it just means that things will have the potential to get messy when you try to port it to 64-bit when you must hand-compute everything and provide different definitions.

Some native libraries may have no issue (though very rare).  If some theoretical library had the same exact definitions/offsets/alignments/packing.....in both 32/64-bit for everything....then nothing would need to change.  However, I wouldn't count on this being the case:)

On Friday, February 5, 2016 at 1:19:36 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

I have v8.6.2 but haven't installed it yet.  Do you know if there are any changes to the DLLs and VM that come with v8.6.1 and v8.6.2?  I ask because if there are no differences I will install v8.6.2 and use it for all my other work.  When there are changes to our product, I just deliver a new image.  The DLLs and VM are not replaced at the user site.  If they need to be, things get more complicated.

That said I would like to move to v8.6.2 and if that makes our life easier for this LibUSB project, all the better.  If you get back to me soon on this, I will see about moving to v8.6.2.

Lou

On Friday, February 5, 2016 at 1:06:48 PM UTC-5, Seth Berman wrote:

Hi Lou,

If you have 8.6.2 image, an example of the new OSStructure capability is in MZStream (and basically everywhere else that was impacted by 64-bit).
On the class side...you see initializeAfterLoad.  However, you see that fixedSize is mysteriously not there.  Instead, you see the member names as normal, but this is followed by data type descriptors for each member name.

Notice the large number of #pointer fields.  This would be incredibly tedious to do by hand and cover 32/64 bit...and so it's not.  You just provide one definition, and then on the instance side you present the member name instead of the offset (because the offset would be different in 32/64-bit).
We technically have the information to figure out the datatype in the accessor on the instance side...but we chose not to do this because it's an extra level of indirection in what is a performance-critical area.  In the future we may provide that capability.

This framework also facilitates embedded or anonymous structures...one of the datatypes specified on #initializeOnLoad can be the name of another OSStructure....in which case it handles all offset/alignment/size computations.

There is some updated documentation for 8.6.2 in the Programmers Reference -> VA Smalltalk Virtual Machine API -> OSObject -> OSStructure section to describe this.
You can take a look here: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/stpr451.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dpr%2Fstpr451.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHRH4qRK74AB749M5LbXtb7q30pCA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dpr%2Fstpr451.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHRH4qRK74AB749M5LbXtb7q30pCA&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/stpr451.html

-- Seth

On Friday, February 5, 2016 at 12:51:29 PM UTC-5, Seth Berman wrote:

Hello Lou,

Thanks, I've imported and loaded it.

I new I would have to ask these questions...so I will now.

Do you want this to be compatible with < 8.6.2 images?

If so, then we must assume we don't have access to the new OSStructure capabilities in 8.6.2 which compute fixedSize, offsets (based on per-platform alignment rules and bitness) for you and permit the instance-size accessors to be reduced to symbolic access instead of hard-coded offsets.

The downside is

1. You would need 8.6.2 or above.

The upside is

1. You don't need separate 32/64-bit definitions or conditionals anywhere.
2. You don't need to hand compute fixedSize or any of the offsets
3. Platform-specific ABI rules are baked into the offset calculator...so you don't need to handle any adjustments to offsets based on platform-specific alignment rule

I'll leave that up to you...some of the defined structs you have are considered opaque from the header file..so you don't really need to model them as OSStructures...they can just be void * as far as your concerned.

The only structures I see that are impacted by 64-bit would be the ones that have pointers in them.  This changes the offsets for any field after that.  The platform alignment rules should probably be ok...these are typically

for double datatypes or the first structure field on AIX...and I don't see, at first glance, any impact in that area.

Let me know how you would like to proceed and I'll try to make any adjustments accordingly.

-- Seth

On Friday, February 5, 2016 at 12:25:42 PM UTC-5, Louis LaBrunda wrote:

Hey Seth,

That's great!!  Attached is version 0.01 of what I have so far.  Kernel is the only prerequisite.  I would like to end up with a package that works with both Windows and Linux.

You will need to get a copy of the LibUSB dll (I'm using libusb-1.0.dll that I think I got here: <a href="http://sourceforge.net/projects/libusb/files/libusb-1.0/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;">project libusb).  In the Pragma I name the library LibUSB.  I put the dll in the folder with my test image and point to it with: LibUSB=libusb-1.0.dll in the [PlatformLibrary Name Mappings] section if the Ini file.

If you execute "KscLibUsbApp getUsbDeviceDescriptors inspect" if it works you will get an array of device descriptors.  Lately I have been having problems with libusb_init.  I will keep playing and send another post when I figure things out or give up and ask you a question (if I can think of the right one).  For now I want to get this posted. 

All of the classes start with "KscLibUsb", I'm open to changing that by maybe removing the "Ksc" part if you like.

You are of course free to help in any way you like.  I think going over all the function definitions in the Pragma would be a good place to start.  I used the definitions found here: <a href="http://libusb.sourceforge.net/api-1.0/api.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;">functions, you can use those or any you like.

Do you know if the 64 bit version keeps the function names or add something like "64" to the end?  I ask because I expect the Pragma definitions would be different, with different names we could just put both in the same Pragma, if the names are the same we will need to think of something else, like separate packages.

If you want to make changes yourself, we will have to come up with some way to merge things.  I'm open to suggestions.

Lou

On Thursday, February 4, 2016 at 4:24:43 PM UTC-5, Seth Berman wrote:

Hi Lou,

I'd be happy to help...send it along.

-- Seth

On Thursday, February 4, 2016 at 4:13:28 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

Thanks for the offer, I will take you up on it.  I have about 100 functions defined in a Pragma and about 20 structures defined.  I have tested and have working LibUsbErrorName, LibUsbGetDeviceList, LibUsbGetDeviceDescriptor, LibUsbStrError, LibUsbGetVersion and some smalltalk methods that make calling the LibUsb functions easier.

I think most of the functions defined in the Pragma are correct.  I'm not sure about those that take structures as input or that fill in structures.  I'm less confident about the definition of the structures.  Especially those that have structures in the structures.  I also find different documentation for some structures, some are longer than others.  I'm not sure how to handle that.

I'm more than willing to do the work but it would be a big help if someone could check what I have so far.  And give me some help where my knowledge of C falls short.

Lou

Hi Seth,

Quick question...what version of libusb are you using to bind to?  There are certain things you have that look like this is a binding to a very old version.

I'm using the 1.0.20 version released in September 2015.

On Sunday, February 7, 2016 at 5:55:47 PM UTC-5, Seth Berman wrote:

Hello Lou,

Ok...try this.  This binds to libusb-1.0.20.  You can get the binaries here: <a href="http://libusb.info/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.info%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGNoefsKZL8RRg77a4Z7dFrAfyqgw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.info%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGNoefsKZL8RRg77a4Z7dFrAfyqgw&#39;;return true;">http://libusb.info/ (Downloads -> Latest Windows Binaries).

I'm using the DLL from the MS32 folder.  As for the structures I was building, I may have started with other .h files and jumped around the Internet looking for definitions, so I may well have a hodgepodge.  Hopefully what you have done replaces the mess.

 

The header file is fully wrapped and you can run KscLibUsbApp examplePrintBasicDeviceAttributes to try out basic functionality.

All functions from the libusb.h header file just need to be referenced from the appropriate OSObjects or Application-level object.

For example, libusb_free_bos_descriptor() should be called from KscLibUsbBosDescriptor>>free.

Make sense.

 

We also need to hookup the callbacks -> EsEntryPoints so we can get notification during actions like transfer and when someone plugs/unplugs a usb device.

Great idea.

 

Most of these structs are pretty low-level concepts...therefore..I would suggest some set of simple interface classes which wrap/delegate to 1..n of these structs.

For example, I could see there would be a UsbDevice object and a UsbTransfer object.  I would either put those in a separate application or have the top-level app be the interfaces and create a subapp to hold the OSObjects.

Those are some ideas for you.

Okay, I will think about that once I'm up on v8.6.2 and I catch up to you.

 

Your context thoughts sound right...

Good.

 

it may be easier if you are going to do a simple set of interface classes, just to have that managed there rather than in the OSObjects.

Okay, I expect you are correct, just want to think about it once I cann see the big picture.

 

You may want to consider moving this up to Vast Goodies to facilitate exchanges?

Will do.

Lou

 

-- Seth

On Saturday, February 6, 2016 at 2:18:09 PM UTC-5, Louis LaBrunda wrote:

Hey Seth,

Thanks for all the info on v8.6.2 and the forthcoming 64 bit VM.

After looking at your version of #usbContext, I think it is not a very good idea to lazy initialize UsbContext.  Instead I propose we have two methods #initDefaultContext and #initNewContext.  #usbContext would just return UsbContext.  #initNewContext will do much of what you have in #usbContext.  #initDefaultContext will call libusb_init with a null pointer.  Both will set UsbContext and return what ever libusb_init returns.  This way we don't have to deal with any errors, we just pass them back to the user.  Methods like my #getUsbDeviceDescriptors can check to see if there is a context and if not call #initDefaultContext.

Can you check to see if #KscLibUsbContext is sub classed from the right class.

Okay, Monday I upgrade to v8.6.2.  I look forward to your LibUSB structures.

Lou

On Friday, February 5, 2016 at 6:00:46 PM UTC-5, Seth Berman wrote:

Hi Lou,

Ok, I see.  So there will always be a 32-bit and a 64-bit image.  The issues are almost innumerable...but lets take something simple.  Objects.  In the 32-bit image a (normal) object will have a 12-byte header followed by 4 byte slots.  In a 64-bit image, this will be a 16-byte header followed by 8 byte slots.  So you can't have a single image, which consists of objects (and other things of course) that could load on both with no modification.  The good news is that we have an on-the-fly image translator that, on the 64-bit vm, notices that the image is 32-bit and converts everything to it's 64-bit format.  Then you save the image and your done...you now have a 64-bit image snapshot of your 32-bit image.  It does not go both ways...you can't load a 64-bit image on a 32-bit vm.  So I imagine the flow will be to develop in your 32-bit image...and when your ready...save the image and load in on the 64-bit vm...and then save it.  This is useful for packaged images.  You could also just load the same code from ENVY in both 32 and 64-bit images and that would work too.

I'll review the pragmas...from what I saw the formats look fine.  I have a similar layout in the new OpenSSL additions.

For libusb_init...this is one of those apis that wants the address of the pointer...or a pointer to a pointer which it will fill in.

In c you typically see this as 'libusb_context *ptr;   libusb_init(&ptr).  This &ptr is a libusb_context ** ...but is stack allocated pointer to what will be a heap allocated pointer libusb will create. We can do the same but will heap allocate the & part.

I haven't tried this yet since I have to get the libusb library...but give this a shot.

usbContext

"Answer our UsbContext."

UsbContext isNil

ifTrue: [| contextPtrPtr |

contextPtrPtr := OSObjectPointer calloc: 1 itemType: KscLibUsbContext.

(LibUsbInit callWith: contextPtrPtr) = 0

ifTrue: [UsbContext := contextPtrPtr at: 0]

ifFalse: [ "Explicit Fail Condition?" ].

contextPtrPtr free "Free the pointer we created....not the one inside that is now pointed to by UsbContext"].

^UsbContext

On Friday, February 5, 2016 at 4:55:16 PM UTC-5, Louis LaBrunda wrote:

Hey Seth,

Sorry about that unclear question.  I was just wondering what I had to do to develop both 32 bit and 64 bit programs once the 64 bit version comes out.  Do I need to keep both 32 bit and 64 bit development environments around?  Anyway, this doesn't effect what we are about to work on.  It is getting late now, so starting Monday I will move to v8.6.2 and we can go from there.

Compared to what I'm used to, there are not that many structures in libusb....so I'd be happy to do an 8.6.2 version of it and pass the changes back for you to see what I'm talking about.

That would be great!!!  Will I need to do much to the Pragma?  Not that I mind.  Is there a way that the functions defined in the Pragma will match up with what the DLL functions expect?  If there is, that would also great as I won't have to guess what type/size parameters are.

Lou

P.S.  There is a second Pragma with a lot of constants defined in it.  Some of them define the sizes of things, from what you tell me, I think some of that can go away.

P.P.S.  I'm having trouble with "libusb_init" if I pass it a null pointer it works fine as it defines a default context.  If I try to pass it a pointer that it can answer/fill-in a pointer/context value I somehow mess up and can't use the pointer with libusb_exit.  I don't know how important it is to support multiple contexts but I think it would be nice to do so and I'm probably just using the wrong pointer class or using it in the wrong way.

On Friday, February 5, 2016 at 4:27:13 PM UTC-5, Seth Berman wrote:

Hi Lou,

"Meaning both 32 and 64 bit versions of the same program and 32 bit versions of one program and 32/64 bit version of another."

I don't follow...can you clarify?

Are you asking the difference between developing 32/64-bit compatible code using the old way vs the new way?

In any case I would say the following

1. If you have an existing 32-bit binding developed and you don't care about using it on 64-bit....then the solution is simple....do nothing.
   - We are not removing the old OSObject API...even for future releases...so it will continue to work on 32-bit vms

2. If you have an existing 32-bit binding developed and you DO care about using it on 64-bit...then you have 2 options.

•    Option 1: Continue to use hard-coded offset based APIs.
  
•  This means you will either carry around with you 2 versions of OSObjects for ones that are impacted by 64-bit or attempt to have one with a bunch of is64BitPlatform conditional checks in it...and even then you need an image that understands System is64BitPlatform (though I guess you could technically query if #is64BitPlatform is available and assume 32-bit if not....though I'll foreshadow the rest of this....we are going down a very bad path...but I'll continue)
  
• You must then figure out how to load / reference the right definition based on which vm you launched the image on.  If you want to save a 32-bit image and start it on a 64-bit image...this means you needs something dynamic....config rules won't cut it.
• You will need to hand compute any changes that impact fixed size and offsets such as field datatype (i.e. pointers), field alignment, structure alignment, packing specification, differences in struct/union definitions between 32/64-bit, and platform dependent ABI rules (i.e. default 8-byte alignment of doubles on Linux vs 4 on Windows).
• This gets more complicated when nested structures or unions are involved.
• If this is the way you want to go, I would not even attempt to figure this out by hand.  I would build a small C program that interfaces with the DLL in question and start asking sizeof and alignmentof questions in both 32/64-bit compiled version so you make sure you got the right answer to the fixedSize and member offsets.
• FYI, I did this for about 5 of the 100s (maybe 1000s) of classes I had to update for the image before I realized I wasn't going to do this....ever.

• Options 2: Use the new API which auto-calculates offset and fixed size for each platform and auto-configures on startup based on the OS and bitness of the vm it was loaded on.
• For structures, this means changing #initializeOnLoad by removing the fixedSize hard coded value (since it does it for you) and using the extended OSStructure #members:types: api to assign the members (which will already be done) and types that mirror the C code field datatypes.
• On the instance side, remove the hard-coded offset number and replace it with the symbolic field name <Symbol> instead.  So instead of 'self uint8At: 24'...it would become 'self uint8At: #fieldName'
• Unions are now first class objects with their own offset/size calculator baked in
• Nested structures are easy...just add the nested OSStructure subclass name to the types portion of the #members:types: API (examples in online doc)
• Anonymous Structures/Unions don't need to be explicitly modeled...they can be defined within the #members:types: definition (examples in online doc)
• In 8.6.2 there are a ton of examples of this.  The 64-bit impacted windows API has been re-wrapped using this and it had tons of obscure differences between 32/64 bit definitions that would have made it a nightmare to port.  In fact, I'm sure I'll still be doing it today:)
• I would look at the documentation too for OSStructures...it covers most all the use cases.  If not...I would be happy to update it

What I'm not covering here, but equally important for 64-bit clean code, is the identification of code that just assumed pointers were 4 bytes.  I've seen everything from 'ByteArray new: 4' to store a pointer, OSUInt32 for returning or passing pointers, (temp uint32At: 0) for retrieving pointers, and even more indirect, harder to spot, ones like when windows passes in the lParam in an OSWidget callback...all the upper 32-bits, of the now 64-bits, are 1s leaving you with an incorrect value when you bitshift away the lower 16 bits to get what you thought was the upper 16 bits.

It's an unfortunate series of changes that have to be made, but that's the way things are.  The OSObject additions really help a lot, but the 'hardness' depends on both the native library and how the binding code was originally coded in Smalltalk.
Sometimes it goes really well, for example our database guy did the coding for SQLite...and I literally only had to make 1 change to make it 64-bit compatible....its nice when things work out that way.

So...maybe a little long winded...but best to get the information out there now for folks to plan for.

To your other question...no...you don't need to do anything special when loading your open-edition apps.  Unless released, any open edition classes will need to be reloaded since only the released versions of the classes will be loaded when you load the open-editition application, but that's always been the case.

Its just important to know that if you save a method in 8.6.2, and then go and look at that method in an 8.6.1 image, for example...then it will have strange characters in the 8.6.1 editor.  This means you didn't copy over your 8.6.2 library primitives to your older image installation.

Compared to what I'm used to, there are not that many structures in libusb....so I'd be happy to do an 8.6.2 version of it and pass the changes back for you to see what I'm talking about.

-- Seth

On Friday, February 5, 2016 at 3:02:28 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

I currently don't have anything deployed that uses OpenSSL, so I don't think my deployment will be impacted by moving to v8.6.2.

I do have one program that uses a fairly old DLL (RSC/Piccolo used to communicate with Tandem/HP NonStop computers).  My interface to this DLL is also very old and was built using the C External Interface parts (AbtCExternalFunction) that are dropped on the white space of a part and the function information entered into the dropped part.  From time to time I have to make changes to this program but not to the parts that deal with the DLL.  This program doesn't need to be 64 bit.  Most, if not all of my programs don't need to be 64 bit but it would probably be good to go in that direction.  How hard would it be to develop for both 32 and 64 bit?  Meaning both 32 and 64 bit versions of the same program and 32 bit versions of one program and 32/64 bit version of another.

Anyway, back to the question of moving to v8.6.2, do I have to version all open apps before I upgrade to v8.6.2?  I have separate images for each app.  Some have open apps with small changes.

Lou

On Friday, February 5, 2016 at 2:12:14 PM UTC-5, Seth Berman wrote:

Hello Lou,

There are some changes to DLLs in 8.6.2...the ones that would impact deployment are the changes to OpenSSL.

Those are referenced here in the migration guide: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8617.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8617.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEAx1F9J7kGizdjhLihB6owhzuqQw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8617.html\46sa\75D\46sntz\0751\46usg\75AFQjCNEAx1F9J7kGizdjhLihB6owhzuqQw&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8617.html

There are new envy library primitives for 8.6.2 for development images.  It probably doesn't impact your deployment, but for your development use with the ENVY repository...it requires some understanding.
Those are referenced here in the migration guide: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8612.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8612.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHN5RjN4Fv0mhkEkQ-NNTjoZmKkkg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dmi%2Fmigr8612.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHN5RjN4Fv0mhkEkQ-NNTjoZmKkkg&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=mi/migr8612.html

In short, the code you develop using the new library primitives (shipped with >= 8.6.2) will NOT be properly readable by images using the older library primitives ( < 8.6.2).
If your running a mixed environment, then the new library primitives must be copied to older versions on the product to use so they can properly decompress the >= 8.6.2 source code.

The importance of keeping as current as possible comes with your desire to use the 64-bit vm when it is released.  The 64-bit vm will definitely NOT run images older than 8.6.2 and never will.

Any third-party native library code not developed by us, and that would be impacted by 64-bit, would also need to be updated.  This is unavoidable, as the 32 to 64-bit image translator can not possibly have

knowledge of how some arbitrary 3rd party DLL is impacted by 64-bit and magically update the OSObjects.  This is why it is preferential to switch to the new OSObject capabilities sooner, rather than later, if you

have interest in running the 64-bit vm.  It reconfigures itself on startup based on the bitness so you only have 1 definition of an OSObject to support both 32 and 64-bit.

There is no doubt that the new capability makes life easier in OSObject development because hand computing anything no longer necessary....but as I said...the unavoidable cost is version compatibility.

The old way of doing things is still supported in 8.6.2 and beyond...if you stick to this it just means that things will have the potential to get messy when you try to port it to 64-bit when you must hand-compute everything and provide different definitions.

Some native libraries may have no issue (though very rare).  If some theoretical library had the same exact definitions/offsets/alignments/packing.....in both 32/64-bit for everything....then nothing would need to change.  However, I wouldn't count on this being the case:)

On Friday, February 5, 2016 at 1:19:36 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

I have v8.6.2 but haven't installed it yet.  Do you know if there are any changes to the DLLs and VM that come with v8.6.1 and v8.6.2?  I ask because if there are no differences I will install v8.6.2 and use it for all my other work.  When there are changes to our product, I just deliver a new image.  The DLLs and VM are not replaced at the user site.  If they need to be, things get more complicated.

That said I would like to move to v8.6.2 and if that makes our life easier for this LibUSB project, all the better.  If you get back to me soon on this, I will see about moving to v8.6.2.

Lou

On Friday, February 5, 2016 at 1:06:48 PM UTC-5, Seth Berman wrote:

Hi Lou,

If you have 8.6.2 image, an example of the new OSStructure capability is in MZStream (and basically everywhere else that was impacted by 64-bit).
On the class side...you see initializeAfterLoad.  However, you see that fixedSize is mysteriously not there.  Instead, you see the member names as normal, but this is followed by data type descriptors for each member name.

Notice the large number of #pointer fields.  This would be incredibly tedious to do by hand and cover 32/64 bit...and so it's not.  You just provide one definition, and then on the instance side you present the member name instead of the offset (because the offset would be different in 32/64-bit).
We technically have the information to figure out the datatype in the accessor on the instance side...but we chose not to do this because it's an extra level of indirection in what is a performance-critical area.  In the future we may provide that capability.

This framework also facilitates embedded or anonymous structures...one of the datatypes specified on #initializeOnLoad can be the name of another OSStructure....in which case it handles all offset/alignment/size computations.

There is some updated documentation for 8.6.2 in the Programmers Reference -> VA Smalltalk Virtual Machine API -> OSObject -> OSStructure section to describe this.
You can take a look here: <a href="http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/stpr451.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dpr%2Fstpr451.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHRH4qRK74AB749M5LbXtb7q30pCA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fwww.instantiations.com%2Fdocs%2F862%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dpr%2Fstpr451.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHRH4qRK74AB749M5LbXtb7q30pCA&#39;;return true;">http://www.instantiations.com/docs/862/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/stpr451.html

-- Seth

On Friday, February 5, 2016 at 12:51:29 PM UTC-5, Seth Berman wrote:

Hello Lou,

Thanks, I've imported and loaded it.

I new I would have to ask these questions...so I will now.

Do you want this to be compatible with < 8.6.2 images?

If so, then we must assume we don't have access to the new OSStructure capabilities in 8.6.2 which compute fixedSize, offsets (based on per-platform alignment rules and bitness) for you and permit the instance-size accessors to be reduced to symbolic access instead of hard-coded offsets.

The downside is

1. You would need 8.6.2 or above.

The upside is

1. You don't need separate 32/64-bit definitions or conditionals anywhere.
2. You don't need to hand compute fixedSize or any of the offsets
3. Platform-specific ABI rules are baked into the offset calculator...so you don't need to handle any adjustments to offsets based on platform-specific alignment rule

I'll leave that up to you...some of the defined structs you have are considered opaque from the header file..so you don't really need to model them as OSStructures...they can just be void * as far as your concerned.

The only structures I see that are impacted by 64-bit would be the ones that have pointers in them.  This changes the offsets for any field after that.  The platform alignment rules should probably be ok...these are typically

for double datatypes or the first structure field on AIX...and I don't see, at first glance, any impact in that area.

Let me know how you would like to proceed and I'll try to make any adjustments accordingly.

-- Seth

On Friday, February 5, 2016 at 12:25:42 PM UTC-5, Louis LaBrunda wrote:

Hey Seth,

That's great!!  Attached is version 0.01 of what I have so far.  Kernel is the only prerequisite.  I would like to end up with a package that works with both Windows and Linux.

You will need to get a copy of the LibUSB dll (I'm using libusb-1.0.dll that I think I got here: <a href="http://sourceforge.net/projects/libusb/files/libusb-1.0/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fsourceforge.net%2Fprojects%2Flibusb%2Ffiles%2Flibusb-1.0%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNHN7zjK3om1pUziyd4f508jqJdw3g&#39;;return true;">project libusb).  In the Pragma I name the library LibUSB.  I put the dll in the folder with my test image and point to it with: LibUSB=libusb-1.0.dll in the [PlatformLibrary Name Mappings] section if the Ini file.

If you execute "KscLibUsbApp getUsbDeviceDescriptors inspect" if it works you will get an array of device descriptors.  Lately I have been having problems with libusb_init.  I will keep playing and send another post when I figure things out or give up and ask you a question (if I can think of the right one).  For now I want to get this posted. 

All of the classes start with "KscLibUsb", I'm open to changing that by maybe removing the "Ksc" part if you like.

You are of course free to help in any way you like.  I think going over all the function definitions in the Pragma would be a good place to start.  I used the definitions found here: <a href="http://libusb.sourceforge.net/api-1.0/api.html" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.sourceforge.net%2Fapi-1.0%2Fapi.html\46sa\75D\46sntz\0751\46usg\75AFQjCNGLBFzPqz4hAbTod5iPyL2Si4afWQ&#39;;return true;">functions, you can use those or any you like.

Do you know if the 64 bit version keeps the function names or add something like "64" to the end?  I ask because I expect the Pragma definitions would be different, with different names we could just put both in the same Pragma, if the names are the same we will need to think of something else, like separate packages.

If you want to make changes yourself, we will have to come up with some way to merge things.  I'm open to suggestions.

Lou

On Thursday, February 4, 2016 at 4:24:43 PM UTC-5, Seth Berman wrote:

Hi Lou,

I'd be happy to help...send it along.

-- Seth

On Thursday, February 4, 2016 at 4:13:28 PM UTC-5, Louis LaBrunda wrote:

Hi Seth,

Thanks for the offer, I will take you up on it.  I have about 100 functions defined in a Pragma and about 20 structures defined.  I have tested and have working LibUsbErrorName, LibUsbGetDeviceList, LibUsbGetDeviceDescriptor, LibUsbStrError, LibUsbGetVersion and some smalltalk methods that make calling the LibUsb functions easier.

I think most of the functions defined in the Pragma are correct.  I'm not sure about those that take structures as input or that fill in structures.  I'm less confident about the definition of the structures.  Especially those that have structures in the structures.  I also find different documentation for some structures, some are longer than others.  I'm not sure how to handle that.

I'm more than willing to do the work but it would be a big help if someone could check what I have so far.  And give me some help where my knowledge of C falls short.

Lou

Thanks, both of you, for the detailed write ups.

On Thursday, February 11, 2016 at 12:13:31 PM UTC-8, Seth Berman wrote:

Hello All,

Lou...this was a great overview of the process.  I hope people find their way here if they are wondering how to get started with developing bindings to C libraries with VA Smalltalk (...and I'm saying this verbosely since this will be google indexed:)

It was excellent! I am somewhat familiar with the process of interfacing to a library. I was looking for what you learn as you go through the process, the little details, the "gotchas", etc. :-)

One of these days, I will have to write up the work I did to make VA support the C function call style of VW. (A subset, actually. Just what was needed to be able to use the VisualWorks code for GBS to interface to the GCI DLL.)
 

Making member access to OSObjects even easier...(i.e. getters/setters)

Lou is right...this can be even easier.  In fact, it can be reduced even to the point where you don't need explicit setters/getters.  Imagine a special subclass called OSReflectiveStructure that has a doesNotUnderstand: handler on it.  It can look at the message and determine if this is a member getter or setter.  If not, then pass it along...if so, then do all the work to figure out what to return.

I hope you don't. Performance aside, there are the issues of clarity, discoverability, and verifiability. With explicit methods, you can find the implementors of sent messages. That also affects Scintilla's highlighting, although you can probably "guess around" that. Being able to connect the dots is important, so being able to find the actual method doing the work helps. And seeing what the method does helps verify that everything is correct.

Generated accessors with offsets in class variables or something like that works better for me.

The reason that this was not done in 8.6.2 comes in 3 parts.  Performance, Type Ambiguity...and the easy one...8.6.2 Delivery Time:)

Performance:

OSObjects are a performance critical part of the product.  They have vm assistance...and the vm has direct knowledge and assumptions about the shape of an OSObject.  For example, the reference and offsetAndRefType fields...the vm knows exactly where to find those.  It also has a lot of vm assisted primitives for quick access to memory.  When accessing memory....we want to get it to a primitive call as quickly as possible.  More indirection via message sends to get to this point simply boils down to more instructions that must be executed.

In regards to my task of updating every framework we have for 64-bit compatibility, I wanted to be cautious about how much indirection I introduced.  I could tell how much additional time getters/setters via Symbols vs Integers were taking via microbenchmarks...but I could not get a good measure, ahead of time, of what impact this would have had if this additional cost was amortized across the product.  And since providing these nicer types of getters and setters did nothing but add runtime to the existing parts of the product....I chose to keep the getters/setters as is, but change the hard-coded offset for a symbol instead of a very generic access like 'self memberAt: #field'.

I always had plans, and still do, to introduce this.  This is the main reason why I keep the type information around.  Currently, it doesn't really need it after it computes the offsets...but we would need it for these nicer APIs.

For certain bindings, where most all the work is done in C, the additional cost of member access would be a non-issue...so it would be best to start with these...and if you find the cost of certain accesses are heavy...then you can selectively fall-back to the lower level APIs.  That will be the plan.

Type Ambiguity:

When updating the product, I noticed that existing getters/setters were not uniform in the way they wanted to represent certain field types.  For something like a char *, that would end up being a null-terminated string, may be returned by some getters as an OSChar8, others may have returned an OSStringZ....and others may have returned just a generic OSPtr...and so on.  This meant that in my automated tooling to update the getters/setters, I could not easily tell what should be returned.  Something like a membersAt: API would need to make a decision to return this as probably an OSStringZ....however....if I did this for the existing ones...this may have changed the expectations of senders of these accessors and break the product in ways that would be hard to find.  So this was not done...and as I said...for existing bindings this has very little advantage.

8.6.2 Delivery Time:

This one is easy...I ran out of time.

-- Seth

On Thursday, February 11, 2016 at 10:03:03 AM UTC-5, Louis LaBrunda wrote:

Hi Richard,

Seth is better equipped to describe the process than I am.  I will give a quick outline, sorry if it is too simplistic.

These interfaces are usually to DLLs or in Linux SOs.  They are almost always written in C.  As such they have one or more ".h" files that describe the constants, structures and functions of the system.  The task is to map these to Smalltalk.  VA Smalltalk constants can be given reasonable names and defined in Pragmas.  Functions can also be given reasonable names and defined as PlarformFunctions in Pragmas.  The PlarformFunction definition links to the DLL and the name of the function in it and lists and describes the parameters (their C type).  I can usually manage this from the .h file but a real C guy shouldn't have a problem.

Then there are the structures, they are usually sub-classes of OSStructure.  In v8.5.2 defining these is a little easier than it was before.  In older versions you declared the field names (members) and in the setter and getter methods hand coded the offset to the data.  In the new version you declare both the field names and their C type.  This allows setters and getters to reference the member name and it calculates the offset from the C type list.  You still need to define how to convert the data to a Smalltalk instance like:

bcdUSB

"Answer the value of bcdUSB."

^self uint16At: #bcdUSB

and:

bcdUSB: aValue

"Set the value of bcdUSB."

self uint16At: #bcdUSB put: aValue

This is where I run into trouble with things like pointers to pointers and three levels of indirection and such.  If you know both Smalltalk and C, this probably isn't too bad (but it is harder than defining the functions).  It is why I am very grateful for Seth's help (I think he wants to use this with a USB game controller and I want to talk to an X10 home controller).

With both the members and there C types defined, I'm not sure why this:

self uint16At: #bcdUSB put: aValue

couldn't be:

self at: #bcdUSB put: aValue

and have the #at:put: figure out how to convert the value from the defined member name as C type.  That would make defining the setters and getters even easier and could be automated.  Seth may have and answer.

Lou

On Wednesday, February 10, 2016 at 4:05:35 PM UTC-5, Richard Sargent wrote:

On Wednesday, February 10, 2016 at 6:41:19 AM UTC-8, Louis LaBrunda wrote:

Hi Richard,

I'm glad you are interested in this project and will certainly post updates.

I'm less interested in USB, per se, than I am in the process you are working through.
So, I am quite interested in the problems you encounter and their resolutions.

But, do keep us apprised of your progress, too!

 

A version is on VAST Goodies.  It requires VA Smalltalk v8.6.2 because we use features of OSStructure that make them easier to define and work on 32 and 64 bit platforms.  It has OSStructures for all the LibUSB structures.  It has Pragmas that define most relevant constants and almost (if not all) the LibUSB functions.  There is one example (thanks to Seth) and I'm working on another.  It binds to libusb-1.0.20.  You can get the binaries here: <a href="http://libusb.info/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.info%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGNoefsKZL8RRg77a4Z7dFrAfyqgw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Flibusb.info%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGNoefsKZL8RRg77a4Z7dFrAfyqgw&#39;;return true;">http://libusb.info/ (Downloads -> Latest Windows Binaries).  I'm using the DLL from the MS32 folder.  I'm working on 64 bit windows 10 home.  Seth has tested with the new up coming 64 VA Smalltalk VM.  Our interface should work on Linux with the proper Linux version of LibUSB.

For now you need to know a lot C to use the interface.  As time permits we will add more Smalltalk to make it easier.  I intend to upload a new version late Thursday or early Friday.

Lou

On Tuesday, February 9, 2016 at 7:05:08 PM UTC-5, Richard Sargent wrote:

>> Talking via Skype may be easier than all this typing.

That's true. But, I am surely not the only one who is interested in how this progresses. Please post updates. :-)

On Monday, February 8, 2016 at 3:22:03 PM UTC-8, Louis LaBrunda wrote:

Hello Seth,

I'm updated to v8.6.2.  I imported your latest version.  It looks great!!  Although I haven't tested it much I think we have a functional API.

I made the change to usbContext and added initDefaultContext and initNewContext and uploaded the new version to VAST Goodies.

I think it is a good time to do some clean up work.  I put some comments in the #initializeAfterLoad methods of the structures that are probably not needed with the new members:types: way of defining them.  Unless you have an objection, I think I will remove them.  I also started adding getter and setter methods to the structures.  I'm not sure if they are all needed, especially the setters.  Did your magic structure generation take care of all these?

You probably noticed the KscLibUsbApp class variables "UsbContext UsbDevicePointers UsbDeviceDescriptors ".  The idea behind them was to hold on to the context, pointers (from LibUsbGetDeviceList) and descriptors (from LibUsbGetDeviceDescriptor) respectively.  I'm not married to this idea, so I would like to know what you think.  I have done nothing to free the structures, so if we keep these, something should be added to free things when they are no longer needed.

You mentioned:

Most of these structs are pretty low-level concepts...therefore..I would suggest some set of simple interface classes which wrap/delegate to 1..n of these structs.

For example, I could see there would be a UsbDevice object and a UsbTransfer object.

I expect I agree but I'm not sure what these classes would do.  Maybe I will understand better after I play around a bit.  If you have anything in mind, I would love to here it.

Are you on Skype?  You can SkypeMe at: callto://PhotonDemon (I'm not sure the callTo: will work).  Talking via Skype may be easier than all this typing.

Lou

On Wednesday, February 17, 2016 at 6:48:54 AM UTC-8, Louis LaBrunda wrote:

Hi Everyone,

To answer my original question, there is no bug in OSPtr, OSStructure or any of their relatives (at least none that I know of) it was just my lack of C knowledge.

Seth or I will start a new thread soon to describe our experience with building and using the interface to LibUSB.

We will continue to post to this thread about the more general topic of building a VA Smalltalk interface to a DLL.

To that point, we have added instance variables to one of our sub classes of OSStructure.  When you think about it, it is a perfectly normal Smalltalk thing to do but not necessarily obviously given the objects map OS/C structures.  Seth has even added a Class variable to one.  Again, perfectly normal, just saying no reason not to, if it makes sense for your interface.  We also added methods beyond the setters/getters so that users of the interface (like me) don't have to know/remember as much C to go to and from Smalltalk objects to the structures.

Lou

Thanks, Lou. This is the kind of information I was looking for.

Will you provide details about what purpose the extra variables have and how the extra methods help? (I realize this is very specific to the USB project, but I think the information in context will be good to know.)