64bit FFI?
Locked
12
12
 
|
Folks - Has anyone tried porting the FFI to 64 bit? I'm loosely considering using a 64bit VM for our servers but I do need the FFI (mostly for ODBC integration) and I've never looked at any of the ABIs. Alternatively, is it reasonable to try to use 64bit libs from within a 32bit Squeak VM? Thanks for any info! Cheers, - Andreas |
|
|
On Thu, Feb 12, 2009 at 01:32:02AM -0800, Andreas Raab wrote: > > Folks - > > Has anyone tried porting the FFI to 64 bit? I'm loosely considering > using a 64bit VM for our servers but I do need the FFI (mostly for ODBC > integration) and I've never looked at any of the ABIs. Alternatively, is > it reasonable to try to use 64bit libs from within a 32bit Squeak VM? > > Thanks for any info! > > Cheers, > - Andreas Yes, I have done this: http://bugs.squeak.org/view.php?id=7237 http://lists.squeakfoundation.org/pipermail/vm-dev/2008-May/001945.html The changes are fairly extensive, and will require a coordinated update of the platforms and VMMaker sources. I will be happy to help in any way I can. It would be best to do the updates at some time when you, John and Ian are all available to commit the platforms updates at more or less the same time. On the VMMaker side, I can do updates on SqueakSource, and also provide stand-alone changes sets for updating other VMMaker code bases (I think that the necessary change sets are all on Mantis already). It goes without saying that a careful review of my changes would be in order ;) Dave |
|
|
In reply to this post by Andreas.Raab
On Thu, Feb 12, 2009 at 1:32 AM, Andreas Raab <[hidden email]> wrote:
Depends on what one wants to pass through the FFI. On x86-64/EMT64 structure passing conventions are efficient and complex, distributing structure fields across integer and floating-point registers. It is possible to interpret such calls given a type signature for the structure to be passed (I did it for VisualWorks) but it is not easy and it is slow. I would counsel writing a proper ABI compiler in Smalltalk that generates e.g. RTL that is either interpreted or compiled by the VM to make the actual call.
In theory the use of 64-bit libs, provided one has the relevant FFI support should work fine. Smalltalk has the ability to deal with 64-bit and 128-bit values easily. But one does have to write all the support.
To get acquainted with the de facto standard x86-64 ABI read the pages 13 through 22 (section Function Calling Sequence) paying particular attention to the 3 1/2 page algorithm for marshalling :)
and start going to bed earlier ;)
|
|
|
On Thu, Feb 12, 2009 at 9:15 AM, Eliot Miranda <[hidden email]> wrote:
I should expand on this, and could be completely wrong. One would have to use a 64-bit address space, which implies compiling the 32-bit VM using a 64-bit compiler. When DEC and ParcPlace co-implemented 32-bit VisualWorks (my first job on joining ParcPlace was to productise this port) we used the DEC C compiler's ability to define "short" (32-bit) pointers and put the executable in the lower 32-bits of the address space (google for e.g. -xtaso or taso). Hence the VM was a 64-bit executable which had a 32-bit Smalltalk living in the bottom 32-bits. If x86-64 compilers have the same short pointer support then the compilation of a 32-bit Squeak VM should be relatively easy. Otherwise one will have to define the longAt: longAt:put: et al machinery to extend 32-bit addresses into full 64-bit pointers and genuflect to the linker and/or memory allocator to get it to place the heap low in the address space.
|
|
|
On 12.02.2009, at 18:22, Eliot Miranda wrote: > > I should expand on this, and could be completely wrong. Since you're late to the game I'll point out http://squeakvm.org/squeak64/faq.html - Bert - |
|
|
On Thu, Feb 12, 2009 at 9:43 AM, Bert Freudenberg <[hidden email]> wrote:
I see that Bert, but Andreas is asking something different. He is asking if the existing 32-bit system can be run on a 64-bit platform, not that one can run a 64-bit VM on a 64-bit platform. In Qwaq we have 32-bit server images and Andreas is interested in using these images largely unchanged to run as 64-bit processes.
Cheers Eliot
|
|
|
On 12.02.2009, at 18:48, Eliot Miranda wrote: > > > On Thu, Feb 12, 2009 at 9:43 AM, Bert Freudenberg <[hidden email] > > wrote: > > On 12.02.2009, at 18:22, Eliot Miranda wrote: > > I should expand on this, and could be completely wrong. > > Since you're late to the game I'll point out > > http://squeakvm.org/squeak64/faq.html > > I see that Bert, but Andreas is asking something different. He is > asking if the existing 32-bit system can be run on a 64-bit > platform, not that one can run a 64-bit VM on a 64-bit platform. In > Qwaq we have 32-bit server images and Andreas is interested in using > these images largely unchanged to run as 64-bit processes. ... which is what the 64 bit VM does, as Andreas is well-aware. One of the remaining problems is 64 bit FFI, and a few more plugins. Fortunately David has been working on those for quite some time. - Bert - |
|
|
On Thu, Feb 12, 2009 at 9:53 AM, Bert Freudenberg <[hidden email]> wrote:
IIUC the 64-bit VM does _not_ run 32-bit images. So I don't understand how the 64-bit VM is releant. Can yu lead me by the nose a little further? TIA
|
|
|
On 12.02.2009, at 19:00, Eliot Miranda wrote: >>> I see that Bert, but Andreas is asking something different. He is >>> asking if the existing 32-bit system can be run on a 64-bit >>> platform, not that one can run a 64-bit VM on a 64-bit platform. >>> In Qwaq we have 32-bit server images and Andreas is interested in >>> using these images largely unchanged to run as 64-bit processes. >>> >> ... which is what the 64 bit VM does, as Andreas is well-aware. One >> of the remaining problems is 64 bit FFI, and a few more plugins. >> Fortunately David has been working on those for quite some time. > > IIUC the 64-bit VM does _not_ run 32-bit images. It does. > So I don't understand how the 64-bit VM is releant. Can yu lead me > by the nose a little further? Sure. *Please* read the FAQ: > http://squeakvm.org/squeak64/faq.html - Bert - |
|
|
On Thu, Feb 12, 2009 at 10:20 AM, Bert Freudenberg <[hidden email]> wrote:
Um, I have (studied my Agrippa), and I quote: If you try to run a 64-bit image with a VM built for 32-bit images, you will get an error message such as this: This interpreter (vers. 6502) cannot read image file (vers. 68000).If you try to run a 32-bit image using a VM built for 64-bit images, you will get an error message such as this: This interpreter (vers. 68000) cannot read image file (vers. 6502). So no, one cannot run a 32-bit image in a 64-bit VM (as I had thought) and the FAQ is quite specific about it. One has to convert a 32-bit image to a 64-bit image to run it on the 64-bit machine (as detailed ijn the FAQ) and that is not what Andreas is interested in doing.
Apologies for being so persistent :) - Bert - |
|
|
2009/2/12 Eliot Miranda <[hidden email]>: > > > > On Thu, Feb 12, 2009 at 10:20 AM, Bert Freudenberg <[hidden email]> wrote: >> >> >> On 12.02.2009, at 19:00, Eliot Miranda wrote: >>>>> >>>>> I see that Bert, but Andreas is asking something different. He is asking if the existing 32-bit system can be run on a 64-bit platform, not that one can run a 64-bit VM on a 64-bit platform. In Qwaq we have 32-bit server images and Andreas is interested in using these images largely unchanged to run as 64-bit processes. >>>>> >>>> ... which is what the 64 bit VM does, as Andreas is well-aware. One of the remaining problems is 64 bit FFI, and a few more plugins. Fortunately David has been working on those for quite some time. >>> >>> IIUC the 64-bit VM does _not_ run 32-bit images. >> >> It does. >> >>> So I don't understand how the 64-bit VM is releant. Can yu lead me by the nose a little further? >> >> Sure. *Please* read the FAQ: > > Um, I have (studied my Agrippa), and I quote: > Does my 64-bit VM run both 32-bit and 64-bit images? No. Any VM will run either 32-bit or 64-bit images, but not both. You can select one or the other when you generate sources with VMMaker, and you can install both flavors of VM on your system (one each for 32-bit images and 64-bit images). > > If you try to run a 64-bit image with a VM built for 32-bit images, you will get an error message such as this: > > This interpreter (vers. 6502) cannot read image file (vers. 68000). > > If you try to run a 32-bit image using a VM built for 64-bit images, you will get an error message such as this: > > This interpreter (vers. 68000) cannot read image file (vers. 6502). > > So no, one cannot run a 32-bit image in a 64-bit VM (as I had thought) and the FAQ is quite specific about it. On the contrary: Can I run a 32-bit image on my 64-bit computer? Yes. A 32-bit image can be run on either a 32-bit VM or a 64-bit VM. Some computer platforms (e.g. 64-bit Linux) can run both the 32-bit VM and 64-bit VM on the same system. Cheers PHilippe |
|
|
In reply to this post by Eliot Miranda-2
Eliot Miranda wrote: > I see that Bert, but Andreas is asking something different. He is > asking if the existing 32-bit system can be run on a 64-bit platform, > not that one can run a 64-bit VM on a 64-bit platform. In Qwaq we have > 32-bit server images and Andreas is interested in using these images > largely unchanged to run as 64-bit processes. Well, actually, I am asking both ;-) The real problem is elsewhere. We have two issues that we need to address, one is the need for 32 bit libraries on a 64 bit system[*] and the other one is the need for utilization of more memory. The latter is a minor issue though, the real problem is the former. Because of this, I really don't care whether the image is 32 or 64 bit, but I *do* care whether the library the VM links against and that the plugins (FFI above all else) will work in the 64bit environment. So in short, a 64bit VM is a must, a 64bit image somewhat optional. [*] Really, the problem is conflicts between the two. Some libraries cannot have both 32 and 64 bit versions installed on the same box and unfortunately ODBC (which we rely heavily on) is one of those. Cheers, - Andreas |
|
|
In reply to this post by Eliot Miranda-2
Eliot Miranda wrote: > Um, I have (studied my Agrippa), and I quote: > > Does my 64-bit VM run both 32-bit and 64-bit images? > No. Any VM will run either 32-bit or 64-bit images, but not both. This is a bit misleading. What it means is that once you've decided whether a 64 bit VM will be used with 32 or 64 bit images, you cannot change it. However, 64 bit VMs can absolutely be used to run 32 bit images (and why wouldn't they). Cheers, - Andreas > You can select one or the other when you generate sources with > VMMaker, and you can install both flavors of VM on your system (one > each for 32-bit images and 64-bit images). > > If you try to run a 64-bit image with a VM built for 32-bit images, > you will get an error message such as this: > > This interpreter (vers. 6502) cannot read image file (vers. 68000). > > If you try to run a 32-bit image using a VM built for 64-bit images, > you will get an error message such as this: > > This interpreter (vers. 68000) cannot read image file (vers. 6502). > > > So no, one cannot run a 32-bit image in a 64-bit VM (as I had thought) > and the FAQ is quite specific about it. One has to convert a 32-bit > image to a 64-bit image to run it on the 64-bit machine (as detailed ijn > the FAQ) and that is not what Andreas is interested in doing. > > Apologies for being so persistent :) > > > > > http://squeakvm.org/squeak64/faq.html > > > > - Bert - > > |
|
|
In reply to this post by Andreas.Raab
On Thu, Feb 12, 2009 at 10:55 AM, Andreas Raab <[hidden email]> wrote:
That's funny. This is on linux right? I always thought that the linux scheme of keeping the old names /lib, /usr/lib etc 32-bit and adding /lib64 /usr/lib64 was a neat way of allowing old 32-bit binaries to run unchanged. So where is the conflict with the ODBC libraries?
|
|
|
Eliot Miranda wrote: > That's funny. This is on linux right? I always thought that the linux > scheme of keeping the old names /lib, /usr/lib etc 32-bit and adding > /lib64 /usr/lib64 was a neat way of allowing old 32-bit binaries to run > unchanged. So where is the conflict with the ODBC libraries? Ask Chris. Cheers, - Andreas |
|
|
In reply to this post by Andreas.Raab
On Thu, Feb 12, 2009 at 10:59 AM, Andreas Raab <[hidden email]> wrote:
I'm assuming that in-memory the 64-bit VM has 64-bit oops, right? So the VM has to swizzle 32-bit pointers to 64-bit. I haven't come across this swizzling code yet. I thought that the swizzling was done by the system tracer. Where is the in-VM code to swizzle pointers to 64-bits?
|
|
|
Eliot Miranda wrote: > I'm assuming that in-memory the 64-bit VM has 64-bit oops, right? So > the VM has to swizzle 32-bit pointers to 64-bit. I haven't come across > this swizzling code yet. I thought that the swizzling was done by the > system tracer. Where is the in-VM code to swizzle pointers to 64-bits? oopForPointer/pointerForOop translate between OOPs and pointers. So you can translate from a 32bit Oop into a 64bit pointer by adding a base address and vice versa. Cheers, - Andreas |
|
|
On Thu, Feb 12, 2009 at 11:07 AM, Andreas Raab <[hidden email]> wrote:
AFAICT you can produce a 64-bit VM that runs a 32-bit image with this technique, but you can't get a VM that runs both.
- If one fetches 64-bits when converting an oop to an address one can load a 64-bit oop but not a 32-it one, because in fetching 64-bits one will fetch two oops, right? - if one fetches 32-bits when converting an oop to an address one can load a 32-bit oop but not a 64-it one, because in fetching 32-bits one will incorrectly ignore the upper 32-bits of the oop, right? (Unless of course the upper 32-bits of all oops are zero, but that's not a 64-bit image, that's a 32-bit image masquerading as a waste of space).
So I don't see how this allows one to have a 64-bit VM that can run true 64-bit images. What am I missing? On the other hand, start-up code that reads a 32-bit image and swizzles it to a full 64-bit image is perfectly sensible and quite easy in the Squeak context because (IIRC) the 64-bit format doesn't do things like widen the SmallInteger range, implement immediate Floats, etc. Its simply a matter of reading the data, writing two words for each oop and scaling each non-SmallInteger oop appropriately. But as I said I don't see this code anywhere.
David, since you've been working on this perhaps you could clear-up the confusion.
|
|
|
Eliot Miranda wrote: > So I don't see how this allows one to have a 64-bit VM that can run true > 64-bit images. What am I missing? You are mostly confused about the various combinations ;-) Let's start from first principles: You can run 32bit images on top of 64 bit VMs. This is achieved by using oopForPointer/pointerForOop. In order to run 64 bit images you need a 64 bit image to begin with. The 64 bit VM doesn't auto-magically convert your 32bit image into a 64 bit one. This is done by SystemTracer and friends. Does that help? Cheers, - Andreas |
|
|
In reply to this post by Eliot Miranda-2
On 12.02.2009, at 19:29, Eliot Miranda wrote: > Um, I have (studied my Agrippa), and I quote: > > Does my 64-bit VM run both 32-bit and 64-bit images? No. Any VM will > run either 32-bit or 64-bit images Read it like this: Does my 64-bit VM run *both* 32-bit and 64-bit images? No. Any VM will run *either* 32-bit *or* 64-bit images > Apologies for being so persistent :) Well everyone has a thick day once in a while. > http://squeakvm.org/squeak64/faq.html You still fail to see the crucial distinction the FAQ makes between 64- bit *images* and 64-bit *VMs*: What is a 64-bit image? A 64-bit image is an image in which the object memory uses a 64-bit word size for object pointers What is a 64-bit VM? A 64-bit VM is one which is compiled with the LP64 or ILP64 data model. This means, in C terms, that pointers and longs are 64-bits wide. These two are independent. There are four kinds of VMs: 1) a 32-bit VM compiled to run 32-bit images 2) a 32-bit VM compiled to run 64-bit images 3) a 64-bit VM compiled to run 32-bit images 4) a 64-bit VM compiled to run 64-bit images Wether the VM can run 32 or 64 bit *images* is governed by the checkbox in VMMaker. Wether the VM can run on a 32 or 64 bit host is governed by the C compiler. Also, nobody uses 64 bit images yet. Please suggest how the FAQ could make this more clear (assuming you actually studied the whole thing and not just skimmed it). - Bert - |
«
Return to Squeak VM
|
1 view|%1 views
| Free forum by Nabble | Edit this page |