Hi, there.
I happily observe the recent efforts to make 64-bit VMs stable for Linux and Mac OSX. Although Tobias and myself agreed to support the Windows platform in terms of internal and external plugins such as SqueakSSL and FilePlugin, adding 64-bit seems like a lot of work. We have only so much time. Who's out there interested in, thinking about, or already contributing to a 64-bit OpenSmalltalk VM for Windows? Best, Marcel |
I don't promise to spend any time on it, but I've inquired a bit a few months ago. The 1st thing required is to convert a bunch of (int) type declaration into sqint in platforms/win32 because they're not compatible in 64bits.2016-07-05 11:20 GMT+02:00 marcel.taeumel <[hidden email]>:
|
In reply to this post by marcel.taeumel
I’m interested, and I’m moving pharo infrastructure into opensmalltalk-vm infrastructure to be able to collaborate. I’m taking some time, but I will be there sooner than later :) Esteban > On 05 Jul 2016, at 11:20, marcel.taeumel <[hidden email]> wrote: > > > Hi, there. > > I happily observe the recent efforts to make 64-bit VMs stable for Linux and > Mac OSX. > > Although Tobias and myself agreed to support the Windows platform in terms > of internal and external plugins such as SqueakSSL and FilePlugin, adding > 64-bit seems like a lot of work. We have only so much time. > > Who's out there interested in, thinking about, or already contributing to a > 64-bit OpenSmalltalk VM for Windows? > > Best, > Marcel > > > > -- > View this message in context: http://forum.world.st/Poll-Who-is-interested-in-thinking-about-or-already-contributing-to-a-64-bit-OpenSmalltalk-VM-for-Wi-tp4904953.html > Sent from the Squeak VM mailing list archive at Nabble.com. |
In reply to this post by Nicolas Cellier
Hi Nicolas, if we target Cygwin as a build environment, this might be worth noticing: https://cygwin.com/faq.html#faq.programming.64bitporting If we (eventually?) target MS Visual Studio (resp. its C compiler), the code might look different. Not sure. In the Windows (kernel) code, I noticed the use of typedefs, which we could also establish in the vm's windows-specific platform code: https://msdn.microsoft.com/en-us/library/windows/desktop/ff381404(v=vs.85).aspx I wonder if we could manage to write code that compiles in both Cygwin 64-bit and the (free) MS Visual C/C++ compiler: https://www.microsoft.com/en-us/download/details.aspx?id=41151 ... or maybe MS Visual C/C++ only? It would remove Cygwin as a layer of indirection between dev tools and execution platform... *duck-and-run* :-D Overall, it should be really simple (and cheap?) to setup a Windows dev environment for VM developers. Maybe for real, maybe in a VirtualBox only. This helps Eliot and other cross-platform VM developers to debug like they are doing now. Best, Marcel |
2016-07-05 15:40 GMT+02:00 marcel.taeumel <[hidden email]>:
Hi Marcel, if we target Cygwin as a build environment, this might be worth noticing: As currently generated, the Spur Vm for 64 bits expects sizeof(long) == 8. So it is cygwin64 x86_64 compatible, but not so much MSVC... (or mingw-w64 variants...) IMO, this is the easiest target. then we could inquire about alternate compilers. If we (eventually?) target MS Visual Studio (resp. its C compiler), the code I wonder if we could manage to write code that compiles in both Cygwin Yes, it's doable, it's a matter of defining the sq* types and sticking to these types. But that might mean revising VMMaker package to avoid direct references to long/unsigned long/, as well as some of the platforms/* files... ... or maybe MS Visual C/C++ only? It would remove Cygwin as a layer of Using MSVC requires additional support like atomic operations (see ../../platforms/Cross/vm/sqAtomicOps.h) Overall, it should be really simple (and cheap?) to setup a Windows dev Using IDE has a lot of advantages, but maintaining the IDE-specific or even worse IDE-version-specific project files is not sustainable For these reasons, Eliot removed the Xcode projects for Mac, I don't think re-introducing them for windows would be a good idea. Instead, I much much prefer to generate the project files via cmake. The question remains whether we maintain the cmakelists.txt or generate them from Smalltalk (like Pharo VM) I like the idea of a virtual machine prepared for dev, but what about: - license (unless we can redistribute windows 10?) - security (download a virtual machine with unknown installed software, backdoors, etc... ) Best, |
Hi All,
On Tue, Jul 5, 2016 at 8:06 AM, Nicolas Cellier <[hidden email]> wrote:
This is great to hear. So there is a model where sizeof(long) == 8? If that's so, we should target it. There's a /lot/ of work to do if we have to support the Win64 sizeof(long) == 4 model.
There ar emote issues in the type inferencer. It, and a significant ammount of code in VMMaker would have to be rewritten to support 64-bit sizeof(long) == 4. I think it's infeasible given our current person power, and I don't think its wrath it. If we can get there using cygwin ad/or mingw we should do so.
Minor difficulty. MSVC has support for asm. Further, later versions are using clang for their compiler and that *may* just have extended asm support. And do we know that MSVC doesn't support the relevant intrinsics?
+1. Please, something that is either arasllels or can be converted into Parallels.
Why do we need project files? Why do we need cmake? I don't want to keep on fighting this battle, please. Esteban is already working on the issue of generating the header file that generates just the defines that describe the platform facilities. Plugins are selected via plugins.int and plugins.ext. Optional compilation can be controlled with the relevant code in platforms/PLATFORM/plugins/Makefile which can test for available support libraries and abort creation if so. We. Do. Not. Need/ A. CMake. Step. Other, Than. To. Define. The. Platform's. Facilities. I. Do. Not. Want. Project. Files. Under. Any. Circumstances. Please, I thought we had reached agreement on this when we discussed moving to github.
_,,,^..^,,,_ best, Eliot |
On 06.07.2016, at 04:04, Eliot Miranda <[hidden email]> wrote: > > This helps Eliot and other cross-platform VM developers to debug like they > are doing now. > > Using IDE has a lot of advantages, but maintaining the IDE-specific or even worse IDE-version-specific project files is not sustainable > For these reasons, Eliot removed the Xcode projects for Mac, I don't think re-introducing them for windows would be a good idea. > Instead, I much much prefer to generate the project files via cmake. > > Why do we need project files? Why do we need cmake? I don't want to keep on fighting this battle, please. Esteban is already working on the issue of generating the header file that generates just the defines that describe the platform facilities. Plugins are selected via plugins.int and plugins.ext. Optional compilation can be controlled with the relevant code in platforms/PLATFORM/plugins/Makefile which can test for available support libraries and abort creation if so. > > We. Do. Not. Need/ A. CMake. Step. Other, Than. To. Define. The. Platform's. Facilities. > > I. Do. Not. Want. Project. Files. Under. Any. Circumstances. > > Please, I thought we had reached agreement on this when we discussed moving to github. > Dear all, please lets defer that discussion until a later point in time, where we have more energy to discuss that less emotional. We're all invested in adjusting to the github move, I presume so lets focus on that. Let's defer win64 a bit, lets defer MSVC a bit. But lets discuss it eventually, and then, inevitably, we have to re-evaluate a more portable (with regards to build-environments such as VisualStudio,Eclipse,Xcode) build system, be it cmake, scons, gyp, autotools, waf, or whatever. But please lets wait until we are comfortable with the current setup of GitHub, Travis, AppVeyor, bintray, and the like. Let's have a tea or a drink. Best regards -Tobias |
In reply to this post by Eliot Miranda-2
Hi, oh, I did not explain it sufficiently what I mean by saying "dev environment". I did not mean putting project files into the repository or using cmake. No, no, no. I mean a simple document that says how to install and configure Cygwin or Visual Studio to setup the project. There is imo no urgent need to provide ready-made project files. A documentation about paths and compiler flags would be a good first step. Best, Marcel |
2016-07-06 9:04 GMT+02:00 marcel.taeumel <[hidden email]>:
OK my bad, I think i missunderstood. Still, an IDE does boost productivity, especially when navigating from error messages to source code for example when porting win32 to x64... we don't develop in Smalltalk with vi. But let this apart, it's another thread. With a mixture of SourceTree / WinMerge / cygwin bash ./mvm -f ; less LOGF ; grep -r x86_64 ../../platforms / vim / Notepad++ etc... I've got an ersatz of IDE ;) I've cherry picked a few changes required for compiling a X86_64 windows versions and pushed to main cog branch. There are many others waiting, but I don't want to commit a massive blob, so I'll continue to slowly decompose if it's ok. Fortunately the win32 flavour should continue to compile, so I didn't open a separate feature branch (if it lasts too long I fear it would require many rebase and/or merge and that would be boring). Reviews are welcome. cheers
|
In reply to this post by Eliot Miranda-2
2016-07-06 4:04 GMT+02:00 Eliot Miranda <[hidden email]>: Nicolas
snip... After inquiring a bit more, that's not going to be that easy... Yes, x86_64-pc-cygwin-gcc.exe is LP64 and Spur64 compatible, but it lacks all the interfaces used by platforms/win32 (like _fmode _O_BINARY etc...) On the other hand, x86_64-w64-mingw32-gcc.exe has all the required win32 interfaces but is thus LLP64. Thus, either we use cygwin, switch to a unix architecture, and loose a bunch of win32 specific features (Direct3D etc...). But how to provide the interface to window manager/GUI? Or we take the longer path to get VMMaker work on LLP64 (sizeof long == 4, sizeof(long) < sizeof(char *)). |
On Thu, Jul 07, 2016 at 12:45:14PM +0200, Nicolas Cellier wrote: > > 2016-07-06 4:04 GMT+02:00 Eliot Miranda <[hidden email]>: > > > > > Hi All, > > > > On Tue, Jul 5, 2016 at 8:06 AM, Nicolas Cellier < > > [hidden email]> wrote: > > > >> > >> > >> 2016-07-05 15:40 GMT+02:00 marcel.taeumel <[hidden email]>: > >> > >>> > >>> >> > >>> >> -- > >>> >> View this message in context: > >>> >> > >>> http://forum.world.st/Poll-Who-is-interested-in-thinking-about-or-already-contributing-to-a-64-bit-OpenSmalltalk-VM-for-Wi-tp4904953.html > >>> >> Sent from the Squeak VM mailing list archive at Nabble.com. > >>> >> > >>> > >>> Hi Nicolas, > >>> > >>> > >> Hi Marcel, > >> > >> > >>> if we target Cygwin as a build environment, this might be worth noticing: > >>> https://cygwin.com/faq.html#faq.programming.64bitporting > >>> > >>> > >> As currently generated, the Spur Vm for 64 bits expects sizeof(long) == 8. > >> So it is cygwin64 x86_64 compatible, but not so much MSVC... (or > >> mingw-w64 variants...) > >> IMO, this is the easiest target. then we could inquire about alternate > >> compilers. > >> > > > > This is great to hear. So there is a model where sizeof(long) == 8? If > > that's so, we should target it. There's a /lot/ of work to do if we have > > to support the Win64 sizeof(long) == 4 model. > > > snip... > > After inquiring a bit more, that's not going to be that easy... > Yes, x86_64-pc-cygwin-gcc.exe is LP64 and Spur64 compatible, but it lacks > all the interfaces used by platforms/win32 (like _fmode _O_BINARY etc...) > On the other hand, x86_64-w64-mingw32-gcc.exe has all the required win32 > interfaces but is thus LLP64. > > Thus, either we use cygwin, switch to a unix architecture, and loose a > bunch of win32 specific features (Direct3D etc...). But how to provide the > interface to window manager/GUI? Also, on Windows, FilePlugin uses HANDLE rather than (FILE *) for all references to file operations. > Or we take the longer path to get VMMaker work on LLP64 (sizeof long == 4, > sizeof(long) < sizeof(char *)). That would be the right thing to do. Assuming that a long is large enough to hold a pointer is wrong. But it would not be easy or quick to make the changes. Updates to platforms/Cross would be needed, so it affects more than just the win32 tree. Dave |
2016-07-07 14:47 GMT+02:00 David T. Lewis <[hidden email]>:
Hi Dave, I've started the LLP64 compatibility changes on master branch. It's not so hard if we stick to the sq* types. Then there will be a bunch of VMMaker (long) usage to revise. cheers |
In reply to this post by Nicolas Cellier
In the interest of making this switch easier, I've looked into setting up a build env on Windows using MSYS2 + mingw-w64 rather than cygwin (which would also mean a substantial upgrade of gcc version for windows builds to 5.4), here's what I've done so far to get a building (but crashing) vm: - Download and install MSYS2 (https://msys2.github.io) - Run update procedure of base system (update-core + exit, followed by repeated pacman -Suu + exit. Might need to make new shortcuts to msys2_shell.cmd -mingw64 / msys2_shell.cmd -mingw32 after all is said and done) - Install compiler: x86: pacman -S mingw-w64-i686-gcc x64: pacman -S mingw-w64-x86_64-gcc (currently unused) Install tools: make: pacman -S make (yeah, it's needed) git: pacman -S git (for updateSCCSVersions) Install additional packages used by VM + plugins, for corresponding i686 or x86_64 compiler: Haven't determined what's actually needed for a non-crashing build yet. copy build.win32x86 to build.win32x86.MSYS2 folder, as new base for modified build. I chose to run test .mvm's in squeak.cog.spur target directory, so when that's referenced below, the same changes would be needed in other target dirs as well. All "run" steps below in a mingw32 shell; I did however check in mingw64 shell that the combo is LLP on 64-bits. remove cygwin-specific flags removed in new gcc's/ add flags for newer gcc: in build.XXXX/common/Makefile and Makefile.plugin, remove -mno-cygwin params (removed in GCC 4.something, shouldn't have much affect since we're not using cygwin at all) add -mno-stack-arg-probe after -mno-accumulate-outgoing-args (prevents lots of warnings that maccumulate-outgoing-args is needed to probe stack args) Same flag changes needed to Makefiles in platforms\win32\plugins\BochsIA32Plugin (should probably be moved to build.XXXX\somewhere if possible, or -mno-cygwin made a var like EXPORT below that is easy to change; since these files are outside target directory, just removing like I did will affect cygwin builds). remove -lcrtdll from STDLIBS, (lets MinGW link to msvcrtdll.a only, with both there will be duplicate definition errors) comment out line 160, remove comment on 161 in Makefile (aka use EXPORT:=--export-all-symbols) disable SocketPlugin: It had some fun issues redefining stuff, dunno if the subsequent removal of crtdll would have fixed things, but for now, I disabled it. To do so, some source changes were needed to avoid build failures: - add -DNO_NETWORK to DEFS in Makefile (line 146-147) - add #ifndef NO_NETWORK guards around int win32DebugPrintSocketState(void); in platforms\win32\vm\sqWin32Exports.c and platforms\win32\vm\sqWin32Prefs.c - remove SocketPlugin from plugins.ext fix Mpeg3Plugin defining things it assumes is missing on windows, but isn't with mingw64: change #if (... || defined(WIN32)) before NEEDSTRFUNCS in platforms\Cross\plugins\Mpeg3Plugin\libmpeg\changesForSqueak.c to #if (... || (defined(WIN32) && !defined( _WIN32 ))) (could probably be nicer, however _WIN32 is defined on mingw / microsoft compilers, but not on cygwin platform, so works as distinguisher) setup git and update vm versions, so rc compilation doesn't fail: git config --global user.email "[hidden email]" git config --global user.name "John Doe" run /scripts/updateSCCSVersions configure bochs: change --target=i686-pc-mingw32 \ to -target=i686-w64-mingw32 \ in build.win32x86.MSYS2\bochsx86\conf.COG then ./conf.COG and ../../processors/IA32/bochs/makeem from that folder build vm: run .mvm -a in build.win32x86.MSYS2 (with -a to build just the assert version, for speeding things up) VM will now build, but crash after selecting an image file. Time to investigate what additional packages are needed, and/or look at resolving some of the warnings generated during build, there's lots of "implicit definitions" to start from. Cheers, Henry signature.asc (859 bytes) Download Attachment |
2016-07-12 17:37 GMT+02:00 Henrik Johansen <[hidden email]>:
Yes we can do it directly through MSYS+MINGW or load the appropriate MINGW cross compiler packages in CYGWIN64... I think the cygwin64 setup is easier (see the *.yml files for setting up the bot jobs)
Yes it's LLP64.
Yes the makefile are for a really outdated cygwin version (1.4.something). But you can workaround most problems by passing extra arguments to make (via mvm -- EXTRA-MAKE-ARGS) As long as Eliot use this cygwin config, I hesitate to change anything. It would be necessary to test compatibility of changes with legacy cygwin. But this brand is not distributed anymore. With time machines I succeeded into installing a cygwin 1.5, but it was not functional, so I gave up... We ought to convince Eliot that it's time to revise (Hi Eliot ;) ).
I think it's better to use winsock 2 (-lws2_32.lib or something like this)
Hmm I've produced the 32bits VM with i686-w64-mingw32-gcc on cygwin64, and I think I tested it succesfully, but now you give me a doubt. I have to try again. This is how the travis (appveyor...) artefacts are built. On the 64bits side, I've managed to eliminate the warnings about (conversion to/from pointer from/to integer of different size) realted to LLP64, with appropriate changes in VMMaker and platforms files. Still the 64 bits image does not start yet (the VM quits). Nicolas |
Hi Nicolas,
I suppose there are several places in some low-level parts of Cog/Jit where Long and Pointer-Type are used interchangeably. One would have to fix that first. Only then, we can start caring for interfacing the win-api correctly. Best, Marcel |
2016-07-12 21:24 GMT+02:00 marcel.taeumel <[hidden email]>:
Hi Marcel, I've rewiewed all usage of long and unsigned long and fixed that already in VMMaker. But who knows if I didn't forget a place... I've generated the spursrc and spur64src with these changes. I must change the platforms files simultaneously because the angle of attack I choosed is to use the C Compiler warnings. They are very helpful. I'll publish the VMMaker part on http://smalltalkhub.com/mc/nice/NiceVMExperiments/main when I have time (maybe tomorrow). I'll continue to introduce the platforms changes as long as they are compatible with other squeak builds, in a very atomic manner. I prefer this kind of integration to a long lived feature branch. Long lived branches tend to stall, require rebase and as such are difficult to share. They also focus less attention than direct commit to master. It's great to have comments and questions (by the way, thanks Ben, Henrik, Dave, etc...). cheers
|
In reply to this post by Nicolas Cellier
Hi Nicolas,
|
In reply to this post by marcel.taeumel
Hi Marcel, > On Jul 12, 2016, at 12:24 PM, marcel.taeumel <[hidden email]> wrote: > > > Hi Nicolas, > > I suppose there are several places in some low-level parts of Cog/Jit where > Long and Pointer-Type are used interchangeably. One would have to fix that > first. Only then, we can start caring for interfacing the win-api correctly. That's right. Throughout the Cogit is the assumption that sizeof(long) == sizeof(sqInt) == sizeof(void *). That's why I'm hoping we can use a C compiler which obeys this for win64. It is nontrivial to fix, as I've already discussed. Did my words fall on deaf ears and you're proposing to go ahead? > > Best, > Marcel > > > > -- > View this message in context: http://forum.world.st/Poll-Who-is-interested-in-thinking-about-or-already-contributing-to-a-64-bit-OpenSmalltalk-VM-for-Wi-tp4904953p4906337.html > Sent from the Squeak VM mailing list archive at Nabble.com. |
Hi Eliot. Not deaf ears, but neither option seems trivial to me; a large portion of the platform support code on Windows link against platform libraries. Given the choice between having to rewriting all that to use only Cygwin-provided libs*, or changing to a non-cygwin mingw-w64 based LLP build env, and fixing cogit to disambiguate long and void */usqintptr, I'd rather spend what effort I can on helping achieve the latter. The nice thing is the three required parts (platform, cogit, build env) can all be achieved independently, while keeping the 32bit-build stable; there's no explicit requirement to get an LLP build env running before even starting to look at fixing cogit, and only when that works, look at platform integration (ref. Nicolas' comments about warning-based fixes) Cheers, Henry * As well as disallowing all use of natively compiled 64-bit libraries on windows that include a long argument (or, to stay safe, anything not provided by cygwin), and forever more binding the windows 64-bit build to a Cygwin environment/compiler |
In reply to this post by Eliot Miranda-2
2016-07-12 23:23 GMT+02:00 Eliot Miranda <[hidden email]>:
Hi Eliot, for sizeof(long) != sizeof(void *) that's not a big problem from what i see in VMMaker (unless I'm also blind) there are not so many usage of long/unsigned long. longAt() answers a sqInt so it's OK. I've published VMMaker.oscogLLP64-nice.1901 on http://smalltalkhub.com/mc/nice/NiceVMExperiments/main and the list of changes is rather short. Maybe I forgot some places though, it needs consolidation. for sizeof(sqInt) != sizeof(void *) that's really more difficult. There are assumptions that objectMemory == machineMemory and avoidance of oopForPointer() pointerForOop() macros spreaded across Cog/Spur as I understand it. But we ain't gonna need it until we try to run 32bits image on 64bits VM.
|
Free forum by Nabble | Edit this page |