64bit FFI?
Locked
12
12
 
|
On 12.02.2009, at 20:18, Eliot Miranda wrote: > AFAICT you can produce a 64-bit VM that runs a 32-bit image with > this technique, but you can't get a VM that runs both. That's exactly as stated in the FAQ. Any given VM can run *either* 32- bit images *or* 64-bit images. It's a compile-time choice. > On the other hand, start-up code that reads a 32-bit image and > swizzles it to a full 64-bit image is perfectly sensible and quite > easy in the Squeak context because (IIRC) the 64-bit format doesn't > do things like widen the SmallInteger range, implement immediate > Floats, etc. Its simply a matter of reading the data, writing two > words for each oop and scaling each non-SmallInteger oop > appropriately. But as I said I don't see this code anywhere. This is not the job of the VM. You have to manually create a 64 bit image. - Bert - |
|
|
In reply to this post by Andreas.Raab
On Thu, Feb 12, 2009 at 11:24 AM, Andreas Raab <[hidden email]> wrote:
No. Read my description of how I think oopForPointer/pointerForOop has to work and point out to me how one has code that can do both 64-bit and 32-bit oops and I'll be happy. AFAICT the only way to run 32-bit images above 64-bit VMs is to swizzle and I don;t see swizzling code in the VMs but do know there is a swizzling SystemTracer. So yes, I see that one can start with a 32-bit image and derive a 64-bit one
(been there, done that) and see how (but not why) one would swizzle a 32-bit image to a 64-bit image on start-up (its a slow load). But I don;t see how (efficiently) onecreates a 64-bit VM that will run 32-bit and 64-bit images without swizzling.
There's a dearth of comment from Dan and Ian and David who ave actually worked with the 64-bit VM. I'm going to keep quiet on this topic from now on. I've clearly antagonized more than enough people this a.m.
Apologies for the noise.
|
|
|
In reply to this post by Bert Freudenberg
On Thu, Feb 12, 2009 at 11:25 AM, Bert Freudenberg <[hidden email]> wrote:
Speaking through a self-imposed gag let me suggest the FAQ distinguish clearly between a 64-bit VM (meaning a VM that runs 64-bit images that necessarly is compiled as a 64-bit executable) and a 32-bit VM compiled to run as a 64-bit executable. The latter doesn't count as a 64-bit VM for me. 64-bit clean perhaps.
Let me further suggest that 2) is pointless, in that it can't scale beyond the 32-bit address space but uses about 1.5 times the space of a 32-bit image as it tries to do so. i.e. it'll fail a lot earlier than a 32-bit image. Further, if a 64-bit image is larger than will fit in a 32-bit address space one *cant* run it on 2 because it won't fit. However, one can derive a 32-bit image from a 64-bit image that isn't too big using the SystemTracer (don't know if tis has been implemented; we only implemented 32-bit to 64-bit conversion in VsualWorks, uing something akin to the system tracer).
So I would suggest the FAQ explain tat there are really three different VMs, two 32-bit ones and one 64-bit one. One of the 32-bit VMs is compiled to run as a 64-bit executable on 64-bit systems but only runs a 32-bit image. That at least accords with the planet I grew up on :)
|
|
|
Eliot Miranda wrote: > Let me further suggest that 2) is pointless, in that it can't scale > beyond the 32-bit address space but uses about 1.5 times the space of a > 32-bit image as it tries to do so. i.e. it'll fail a lot earlier than a > 32-bit image. Further, if a 64-bit image is larger than will fit in a > 32-bit address space one *cant* run it on 2 because it won't fit. It is not pointless at all. If we were to switch our servers to 64 bit images while allowing customers to deploy on 32bit, we'd be nuts trying to manage both 32 and 64 bit versions of the same image and trying to keep them in sync. Much easier to give them a 32bit VM that can run the very same 64 bit image that we use everywhere. In addition, it is this very combination that allows people without 64bit boxes to actually help in working out the quirks with the 64bit images. I don't think we would have gotten where we are without it. Cheers, - Andreas |
|
|
In reply to this post by Eliot Miranda-2
On 12.02.2009, at 20:40, Eliot Miranda wrote: > >> 1) a 32-bit VM compiled to run 32-bit images >> 2) a 32-bit VM compiled to run 64-bit images >> 3) a 64-bit VM compiled to run 32-bit images >> 4) a 64-bit VM compiled to run 64-bit images > > Speaking through a self-imposed gag let me suggest the FAQ > distinguish clearly between a 64-bit VM (meaning a VM that runs 64- > bit images that necessarly is compiled as a 64-bit executable) and a > 32-bit VM compiled to run as a 64-bit executable. The latter > doesn't count as a 64-bit VM for me. 64-bit clean perhaps. It may not count for you but that's how we have been calling it. Since nobody expects 64 bit images to work "64-bit" VM refers to the host architecture. I'm not quite sure how we could phase out that nomenclature. > Let me further suggest that 2) is pointless, in that it can't scale > beyond the 32-bit address space but uses about 1.5 times the space > of a 32-bit image as it tries to do so. i.e. it'll fail a lot > earlier than a 32-bit image. Further, if a 64-bit image is larger > than will fit in a 32-bit address space one *cant* run it on 2 > because it won't fit. The point of this was not to be used in production but to *create* VM for running a 64 bit image without having to use a 64 bit host. > So I would suggest the FAQ explain tat there are really three > different VMs, two 32-bit ones and one 64-bit one. One of the 32- > bit VMs is compiled to run as a 64-bit executable on 64-bit systems > but only runs a 32-bit image. That at least accords with the planet > I grew up on :) Only if you want to use 64 bit images. Which nobody wants. Yet. - Bert - |
|
|
In reply to this post by Andreas.Raab
On Thu, Feb 12, 2009 at 11:55 AM, Andreas Raab <[hidden email]> wrote:
I respectfully disagree. The VisualWorks 32-bit to 64-bit converter produces something one can deploy immediately. So its not difficult to keep the two in sync. One merely passes the 32-bit image through the converter and fires up the 64-bit image. Saving the 64-bit image can save you a bit of start-up time, so is worth-while doing. Both these processes can be wrapped up in a shell script and run without user intervention. So to keep the 64-bit image in sync with a 32-bit one is easy. So 2) still strikes me as pointless.
In addition, it is this very combination that allows people without 64bit boxes to actually help in working out the quirks with the 64bit images. I don't think we would have gotten where we are without it. I don't want to be sarcastic, but that doesn't seem to be very far. From Bert I get the sense that no-one is actually using 64-bit Squeak images (he just said as much). The Squeak 64-bit image format is not particularly ambitious. It doesn't give one wider SmallInteger range, more identity hash bits, larger objects, more literals in a method or faster floats, or immediate characters. It is as naive a conversion to 64-bits as one can imagine. I don't see that the rather conservative approach taken with the 64-bit VM has produced something particularly compelling. Note that a memory-bound 64-bit Smalltalk application that can't take advantage of 64-bits to speed-up computation (as for example immediate doubles do) simply uses twice the memory bandwidth to accomplish exactly the same thing as a 32-bit application, and so if memory-limited (doesn't fit in cache) can run twice as slow. The only advantage such a 64-bit system has is in being ale to instantiate more objects.
VisualWorks' 64-bit implementation on the other hand provides immediate doubles that have performance only 1/2 that of SmallIntegers, provides 61-bit SmallIntegers and 20-bit id hashes vs 14 bits, much faster at: and at:put:, and is being used by customers using images > 4Gb in production. I think its got a lot further, and it didn't waste effort with 2) in doing so. So that's an existence proof that 2) doesn't help.
Again Im not trying to be offensive or toot my own horn. I'm just trying to be objective. Regards, Eliot
|
|
|
In reply to this post by Andreas.Raab
On Thu, Feb 12, 2009 at 10:55:51AM -0800, Andreas Raab wrote: > > Well, actually, I am asking both ;-) The real problem is elsewhere. We > have two issues that we need to address, one is the need for 32 bit > libraries on a 64 bit system[*] and the other one is the need for > utilization of more memory. The latter is a minor issue though, the real > problem is the former. Because of this, I really don't care whether the > image is 32 or 64 bit, but I *do* care whether the library the VM links > against and that the plugins (FFI above all else) will work in the 64bit > environment. To the best of my knowledge, on 64 bit Linux there is no way for a 64 bit program to call a library compiled for 32 bit. See for example http://forums.amd.com/devblog/blogpost.cfm?threadid=93648&catid=317 It might be possible to extend the FFI implementation in some way that would allow a 64 bit VM to set up the call stack for a 32 bit library, but I for one certainly do not know how to do this. > So in short, a 64bit VM is a must, a 64bit image somewhat optional. > > [*] Really, the problem is conflicts between the two. Some libraries > cannot have both 32 and 64 bit versions installed on the same box and > unfortunately ODBC (which we rely heavily on) is one of those. If you just need a 64 bit VM calling various 64 bit libraries, then the problem is straightforward, and you (or we) just need to do Mantis 7237. If you also need to call some 32 bit libraries, then the problem is more challenging, and I do not know of a solution (*). Dave (*) If a vendor cannot or will not support their 32 bit and 64 bit ODBC libraries on a single host, I would look for some creative unix hacker to think of a workaround. There is sure to be some way to rename things or chroot or otherwise trick the system into doing what you want, even if the vendor is obstinate or clueless. |
|
|
In reply to this post by Eliot Miranda-2
On Thu, Feb 12, 2009 at 12:18:42PM -0800, Eliot Miranda wrote: > > I don't want to be sarcastic, but that doesn't seem to be very far. From > Bert I get the sense that no-one is actually using 64-bit Squeak images (he > just said as much). The Squeak 64-bit image format is not particularly > ambitious. It doesn't give one wider SmallInteger range, more identity hash > bits, larger objects, more literals in a method or faster floats, or > immediate characters. It is as naive a conversion to 64-bits as one can > imagine. I don't see that the rather conservative approach taken with the > 64-bit VM has produced something particularly compelling. Note that a > memory-bound 64-bit Smalltalk application that can't take advantage of > 64-bits to speed-up computation (as for example immediate doubles do) simply > uses twice the memory bandwidth to accomplish exactly the same thing as a > 32-bit application, and so if memory-limited (doesn't fit in cache) can run > twice as slow. The only advantage such a 64-bit system has is in being ale > to instantiate more objects. The 64 bit image was a research project that I believe was intentionally done in the simplest way possible. Your observations are correct, but your expectations may be a bit high. The objective was to get it working, and the "compelling" part was left for whomever might choose to take up the initial work and do something further with it. You're quite right that the 64 bit image is as naive a conversion to 64 bits as one can imagine, but this was entirely intentional and the folks who made that design choice were not being naive about it in the least. From my point of view, the really nice thing about having the naive implementation available is that, well, it works. Even an amateur like me can pick it up, figure it out, and maybe make a few useful contributions along the way. Worst case I might learn something while I'm at it ;) Dave |
|
|
In reply to this post by Eliot Miranda-2
On Feb 12, 2009, at 11:30 AM, Eliot Miranda wrote: > There's a dearth of comment from Dan and Ian and David who ave > actually worked with the 64-bit VM. Well, Bert and Andreas and Dave are explaining it admirably well. Anything I might add would be redundant at best and confusing at worst. Of course any explanation that makes perfect sense to me might not make any sense whatsoever to someone who is not already in possession of the knowledge that they're trying to communicate. Cheers, Ian |
Universal VM for 32/64 bit images (was: 64bit FFI?)
|
|
In reply to this post by Eliot Miranda-2
On Thu, Feb 12, 2009 at 11:18:02AM -0800, Eliot Miranda wrote: > > AFAICT you can produce a 64-bit VM that runs a 32-bit image with this > technique, but you can't get a VM that runs both. Actually, I think that it probably *is* possible to make a Squeak VM that will run both 32 and 64 bit images, although performance may be poor and the plugins would be problematic. The basic idea is to move BytesPerWord and BaseHeaderSize into run time variables rather than compile time macros (i.e. move them from class variables in ObjectMemory to instance variables in ObjectMemory), and set the values at image load time. Everything else should just work. I've been meaning to do this for quite some time but have not gotten around to trying it. Dave |
Re: Universal VM for 32/64 bit images (was: 64bit FFI?)
|
|
On Thu, Feb 12, 2009 at 7:38 PM, David T. Lewis <[hidden email]> wrote:
You'll have to change oopForPointer/pointerForOop to account for the different widths of an oop. e.g. if the heap is origined at 0 then in a 64-bit image the oop 4 maps to byte address 8, but in a 32-bit image it maps to byte address 4.
You'll have to change the readers and writers (e.g. oopAtPointer/oopAtPointerPut) to test BytesPerWord and fetch/store 64 or 32 bits as required. At least. It won't be particularly sprightly :) I've been meaning to do this for quite some time but have not gotten |
Re: Universal VM for 32/64 bit images (was: 64bit FFI?)
|
|
On Thu, Feb 12, 2009 at 09:47:44PM -0800, Eliot Miranda wrote: > > On Thu, Feb 12, 2009 at 7:38 PM, David T. Lewis <[hidden email]> wrote: > > > > > On Thu, Feb 12, 2009 at 11:18:02AM -0800, Eliot Miranda wrote: > > > > > > AFAICT you can produce a 64-bit VM that runs a 32-bit image with this > > > technique, but you can't get a VM that runs both. > > > > Actually, I think that it probably *is* possible to make a Squeak > > VM that will run both 32 and 64 bit images, although performance may > > be poor and the plugins would be problematic. The basic idea is to > > move BytesPerWord and BaseHeaderSize into run time variables rather > > than compile time macros (i.e. move them from class variables in > > ObjectMemory to instance variables in ObjectMemory), and set the > > values at image load time. Everything else should just work. > > > You'll have to change oopForPointer/pointerForOop to account for the > different widths of an oop. e.g. if the heap is origined at 0 then in a > 64-bit image the oop 4 maps to byte address 8, but in a 32-bit image it maps > to byte address 4. > > You'll have to change the readers and writers > (e.g. oopAtPointer/oopAtPointerPut) to test BytesPerWord and fetch/store 64 > or 32 bits as required. Piece o' cake. I bet you a virtual beer I can get it working without touching any of these ;) > It won't be particularly sprightly :) No wagers on that one. It would be interesting to see just how much slower it is though. Just to hazard a guess, I would expect about a 30% hit. If I get it working, I'll report back on results. Dave |
Re: Universal VM for 32/64 bit images (was: 64bit FFI?)
|
|
In reply to this post by Eliot Miranda-2
Eliot Miranda writes: > On Thu, Feb 12, 2009 at 7:38 PM, David T. Lewis <[hidden email]> wrote: > > > > > On Thu, Feb 12, 2009 at 11:18:02AM -0800, Eliot Miranda wrote: > > > > > > AFAICT you can produce a 64-bit VM that runs a 32-bit image with this > > > technique, but you can't get a VM that runs both. > > > > Actually, I think that it probably *is* possible to make a Squeak > > VM that will run both 32 and 64 bit images, although performance may > > be poor and the plugins would be problematic. The basic idea is to > > move BytesPerWord and BaseHeaderSize into run time variables rather > > than compile time macros (i.e. move them from class variables in > > ObjectMemory to instance variables in ObjectMemory), and set the > > values at image load time. Everything else should just work. > > > You'll have to change oopForPointer/pointerForOop to account for the > different widths of an oop. e.g. if the heap is origined at 0 then in a > 64-bit image the oop 4 maps to byte address 8, but in a 32-bit image it maps > to byte address 4. > > You'll have to change the readers and writers > (e.g. oopAtPointer/oopAtPointerPut) to test BytesPerWord and fetch/store 64 > or 32 bits as required. > > At least. > > It won't be particularly sprightly :) Though, for those of us working on JIT compilers the checks will move back to compile time again. Bryce |
Re: Universal VM for 32/64 bit images (was: 64bit FFI?)
|
|
In reply to this post by David T. Lewis
On Fri, Feb 13, 2009 at 09:42:10AM -0500, David T. Lewis wrote: > On Thu, Feb 12, 2009 at 09:47:44PM -0800, Eliot Miranda wrote: > > > > On Thu, Feb 12, 2009 at 7:38 PM, David T. Lewis <[hidden email]> wrote: > > > > > > Actually, I think that it probably *is* possible to make a Squeak > > > VM that will run both 32 and 64 bit images, although performance may > > > be poor and the plugins would be problematic. The basic idea is to > > > move BytesPerWord and BaseHeaderSize into run time variables rather > > > than compile time macros (i.e. move them from class variables in > > > ObjectMemory to instance variables in ObjectMemory), and set the > > > values at image load time. Everything else should just work. > > > > You'll have to change oopForPointer/pointerForOop to account for the > > different widths of an oop. e.g. if the heap is origined at 0 then in a > > 64-bit image the oop 4 maps to byte address 8, but in a 32-bit image it maps > > to byte address 4. > > > > You'll have to change the readers and writers > > (e.g. oopAtPointer/oopAtPointerPut) to test BytesPerWord and fetch/store 64 > > or 32 bits as required. > > Piece o' cake. I bet you a virtual beer I can get it working without touching > any of these ;) Eliot, I owe you a virtual beer. Dave |
«
Return to Squeak VM
|
1 view|%1 views
| Free forum by Nabble | Edit this page |