Squeak/Pharo VM fork history (was: RE: [Pharo-dev] Squeak and Pharo speed differences)
Locked
12
12
Re: Slang for Cuis (WIP, experimental)
 
|
Hunh - I didn't remember that at all; I thought Boris Shingarov had been Mr OS/2-Squeak.There were two OS/2 ports that did non-overlapping things because they were designed with orthogonal goals in mind. Juan's port was a "real PresentationManager/2 application", but it was a real PM/2 *Squeak* in that it opened a PM/2 window and BitBlt'd the Squeak Display to that. Opposite to that, the idea of "real PM/2" to me meant "Native
Widgets" and I wanted to have that in Squeak. The problem was, in
a live system (which is also "truly single-threaded"), how do I
debug the "native-widget" event loop when the PM/2 is tied up by
the BitBlt UI? This meant that Juan's design wouldn't fit: I
needed to somehow have all the Squeak tools but without touching
PM/2. This was enabled by two things: (1) compiling the VM with
Ian's sqXWindow.c, so Squeak would show on an X display while
appearing to OS/2 initially as a headless app talking to the
network, and (2) writing an FFI for the VM -- "native-widget" PM/2
calls were made via that FFI.
I wouldn't be surprised if it still works. But this was never part of the official releases.I very much wanted to merge it upstream into Ian's unix port (the diff was very small). The desire seems weird and silly now, but back then I somehow thought it would make a difference, and I remember I was very frustrated that the merge didn't happen. But it *is* part of the 1.23 distribution on files.squeak.org today.
Oh yes -- Cheese. This is what 3 years later led to the work on Eclipse SWT.
I started with dissecting Slang out of the Cuis VMMaker packages and VMMakerJS (SqeakJS) project into a couple of packages which could be used for generating anything besides VM plugins.You don't need Alien plugins to do VMIL18-simulation-style development of the core VM. For that there is now the Smalltalk implementation of RSP, which I recently added to VMMaker. There is a wide choice how to run the native CPU code: various simulators (gem5 or qemu), or on production silicon under GDB, or on some FPGA prototype under OpenOCD, ... well ... really anything that knows how to speak RSP. This will have a number of far-reaching consequences; this margin is too narrow to explain them, but you can watch a number of my presentations on the subject. Where does this stand right now? VMIL-18-simulation of Cog runs fine over RSP. VMMaker depends on a subtle inaccuracy in the Alien's emulation of instruction atomicity. This will cause VMMaker to fail on off-the-shelf production silicon. But when you modify the CPU to behave like Alien, OSTVM runs just fine (I verified on gem5). So it's a matter of very small fix to the Cogit to match what the stock processor is doing, before we have the full freedom to run anywhere we want.
The larger problem that stops me from bringing in all the rest of
the system (the target-agnostic synthesizer, the
binutils-in-Smalltalk, the symbolic execution engine, the
superoptimizer, etc), is the lack of a usable mother-Smalltalk. OSTVM is hosted in Squeak. With all my love for Squeak, I still can't use it for this kind of work. There are blocker problems with SUnit (causes erratic behavior of Socket code which works fine outside SUnit), so I can't have tests, I resort to launching everything from doits. I have no PetitParser. I have no JSON. I can't use my FFI-to-Z3 bindings. In the middle of working I randomly get unexpected problems with access to source.squeak.org. And so on.
So. Is there an environment where work would not be blocked by such problems? Let's look.
Pharo VM did not only fork the transpiled C code, they also put the Smalltalk VMMaker code in the repository and maintain it there. They do not use the VMMaker Monticello repository if I understandYou must know some magic! something I've been begging the Pharo people for months to tell me the secret where to download that source from. I tried the headless branch, but it doesn't even compile in Pharo. I tried asking on pharo-dev, but I got zero responses since February.
So. The million-dollar question again: Is there *any* environment that works well enough to do OST VM
work in? For the reasons I just explained above, I believe it would be more than just "a good idea". It is critical to the viability of this whole stack. If your project works, it give us an environment enabling further collaboration on OSTVM. Well, I must admit I don't know which way is shorter / more straightforward: this, or making Squeak work. I would be happy either way. I don't care which: I have much bigger fish to fry. -- boris
|
|
|
In reply to this post by K K Subbu
If you look at ftp://ftp.smalltalkconsulting.com/ you will find JMMSqueakBits.sparseimage SqueakBitsLaterInTheDecade.sparseimage squeakBitsSourceFromLastPartOfTheDecade.sparseimage which is work I did on building the MacVMs from the the late 1990 thru to about 2010. Also see https://isqueak.org/HomePage for the VM api documentation I did for the iPhone port in 2010. I'm not sure how far back the pre-osx build stuff goes I recall building for 7.5.5, buy 9.x was the first OS by apple that would run greater than 24 days without crashing... .... John M. McIntosh. Corporate Smalltalk Consulting Ltd https://www.linkedin.com/in/smalltalk ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, May 18, 2020 7:37 AM, K K Subbu <[hidden email]> wrote: > > > On 18/05/20 5:00 pm, karl ramberg wrote: > > > Squeak was made on MacOS 7, 8 or 9 before Apple used Unix. > > Thank you for the correction. AFAIK, the first releases were on MacOS > System 7.0 (68020 or PPC). Ian's Unix ports and Andreas Raab's Wintel > port followed pretty quickly - a tribute to Squeak's portability. > > Regards .. Subbu |
Re: Squeak/Pharo VM fork history
|
|
In reply to this post by K K Subbu
For more background there is a nice paper on the squeak site.
Scanning the paper pretty quickly shows that it was on 68k first, a Duo 230 is mentioned (33 mhz 68030) as well as a PowerMac 8100 (80 - 110 MHz PPC 601). 18 May 2020 16:37 K K Subbu <[hidden email]> wrote:
|
Re: Slang for Cuis (WIP, experimental)
|
|
In reply to this post by Boris Shingarov
Hi Boris, Replying with CC to the Squeak oversight board - we have a meeting scheduled tomorrow, and I would like to put the following topic on our agenda. Let's see if we can to something to help. From Boris Shingarov's message: OSTVM is hosted in Squeak.?? With all my love for Squeak, I still can't use it for this kind of work.?? There are blocker problems with SUnit (causes erratic behavior of Socket code which works fine outside SUnit), so I can't have tests, I resort to launching everything from doits.?? I have no PetitParser.?? I have no JSON.?? I can't use my FFI-to-Z3 bindings.?? In the middle of working I randomly get unexpected problems with access to source.squeak.org.?? And so on. I will not even pretend to understand some of the issues being discussed, but things like SUnit problems and availabity of JSON (for example) are surely things that we can make progress on. Dave On Tue, May 19, 2020 at 01:05:53PM -0400, Boris Shingarov wrote: > > > >Hunh - I didn't remember that at all; I thought Boris Shingarov had > >been Mr OS/2-Squeak. > There were two OS/2 ports that did non-overlapping things because they > were designed with orthogonal goals in mind. > > Juan's port was a "real PresentationManager/2 application", but it was a > real PM/2 *Squeak* in that it opened a PM/2 window and BitBlt'd the > Squeak Display to that. > > Opposite to that, the idea of "real PM/2" to me meant "Native Widgets" > and I wanted to have that in Squeak.?? The problem was, in a live system > (which is also "truly single-threaded"), how do I debug the > "native-widget" event loop when the PM/2 is tied up by the BitBlt UI??? > This meant that Juan's design wouldn't fit: I needed to somehow have all > the Squeak tools but without touching PM/2.?? This was enabled by two > things: (1) compiling the VM with Ian's sqXWindow.c, so Squeak would > show on an X display while appearing to OS/2 initially as a headless app > talking to the network, and (2) writing an FFI for the VM -- > "native-widget" PM/2 calls were made via that FFI. > > > >I wouldn't be surprised if it still works. But this was never part of > >the official releases. > I very much wanted to merge it upstream into Ian's unix port (the diff > was very small).?? The desire seems weird and silly now, but back then I > somehow thought it would make a difference, and I remember I was very > frustrated that the merge didn't happen.?? But it *is* part of the 1.23 > distribution on files.squeak.org today. > > > >Later this was the base of his extremely interesting Cheese project. > > > Oh yes -- Cheese.?? This is what 3 years later led to the work on Eclipse > SWT. > > > >/I started with dissecting Slang out of the Cuis VMMaker packages and > >VMMakerJS (SqeakJS) project into a couple of packages which could be > >used for generating anything besides VM plugins./ > You don't need Alien plugins to do VMIL18-simulation-style development > of the core VM.?? For that there is now the Smalltalk implementation of > RSP, which I recently added to VMMaker.?? There is a wide choice how to > run the native CPU code: various simulators (gem5 or qemu), or on > production silicon under GDB, or on some FPGA prototype under OpenOCD, > ... well ... really anything that knows how to speak RSP.?? This will > have a number of far-reaching consequences; this margin is too narrow to > explain them, but you can watch a number of my presentations on the subject. > > Where does this stand right now? > > VMIL-18-simulation of Cog runs fine over RSP.?? VMMaker depends on a > subtle inaccuracy in the Alien's emulation of instruction atomicity.?? > This will cause VMMaker to fail on off-the-shelf production silicon.?? > But when you modify the CPU to behave like Alien, OSTVM runs just fine > (I verified on gem5).?? So it's a matter of very small fix to the Cogit > to match what the stock processor is doing, before we have the full > freedom to run anywhere we want. > > > The larger problem that stops me from bringing in all the rest of the > system (the target-agnostic synthesizer, the binutils-in-Smalltalk, the > symbolic execution engine, the superoptimizer, etc), is the lack of a > usable mother-Smalltalk. > > OSTVM is hosted in Squeak.?? With all my love for Squeak, I still can't > use it for this kind of work.?? There are blocker problems with SUnit > (causes erratic behavior of Socket code which works fine outside SUnit), > so I can't have tests, I resort to launching everything from doits.?? I > have no PetitParser.?? I have no JSON.?? I can't use my FFI-to-Z3 > bindings.?? In the middle of working I randomly get unexpected problems > with access to source.squeak.org.?? And so on. > > > So. > > Is there an environment where work would not be blocked by such problems? > > Let's look. > > >Pharo VM did not only fork the transpiled C code, they also put the > >Smalltalk VMMaker code in the repository and maintain it there. They > >do not use the VMMaker Monticello repository if I understand > >correctly. So in their repository and workflow, it really is a source > >fork. > You must know some magic!?? something I've been begging the Pharo people > for months to tell me the secret where to download that source from.?? I > tried the */headless/* branch, but it doesn't even compile in Pharo.?? I > tried asking on pharo-dev, but I got zero responses since February. > > > So. > > The million-dollar question again: > > Is there *any* environment that works well enough to do OST VM work in? > > >I started with dissecting Slang out of the Cuis VMMaker packages and > >VMMakerJS (SqeakJS) project into a couple of packages which could be > >used for generating anything besides VM plugins. > > > >Atleast that's my intentiont, I don't yet know of how that project > >idea will pan out, atleast I intend to learn something of how Slang > >works and how it's generated. > > > >It's currently highly expiremental, but I got some of the existing > >VMMaker testscases "green" and so I thought I could share the current > >state of it. Maybe the whole idea of using it outside the VMMaker is > >not so a good idea, maybe, maybe not :-) > > > For the reasons I just explained above, I believe it would be more than > just "a good idea".?? It is critical to the viability of this whole > stack.?? If your project works, it give us an environment enabling > further collaboration on OSTVM.?? Well, I must admit I don't know which > way is shorter / more straightforward: this, or making Squeak work.?? I > would be happy either way.?? I don't care which: I have much bigger fish > to fry. > > -- boris > > |
Re: Slang for Cuis (WIP, experimental)
|
|
Not sure about the Socket/SUnit or FFI-Z3 issues but this JSON works fine in 5.3: JSON - http://www.squeaksource.com/JSON.html MCHttpRepository location: 'http://www.squeaksource.com/JSON' user: '' password: '' And the latest versions (e.g. March 2013) of PetitParser & PetitParserTests on Lukas Rengglis site work too: MCHttpRepository location: 'http://source.lukas-renggli.ch/petit/' user: '' password: '' Hope htis helps Paul David T. Lewis wrote > Hi Boris, > > Replying with CC to the Squeak oversight board - we have a meeting > scheduled > tomorrow, and I would like to put the following topic on our agenda. Let's > see if we can to something to help. From Boris Shingarov's message: > > > OSTVM is hosted in Squeak.?? With all my love for Squeak, I still > can't > use it for this kind of work.?? There are blocker problems with SUnit > (causes erratic behavior of Socket code which works fine outside > SUnit), > so I can't have tests, I resort to launching everything from doits.?? > I > have no PetitParser.?? I have no JSON.?? I can't use my FFI-to-Z3 > bindings.?? In the middle of working I randomly get unexpected > problems > with access to source.squeak.org.?? And so on. > > I will not even pretend to understand some of the issues being discussed, > but things like SUnit problems and availabity of JSON (for example) are > surely things that we can make progress on. > > Dave > > > > On Tue, May 19, 2020 at 01:05:53PM -0400, Boris Shingarov wrote: >> >> >> >Hunh - I didn't remember that at all; I thought Boris Shingarov had >> >been Mr OS/2-Squeak. >> There were two OS/2 ports that did non-overlapping things because they >> were designed with orthogonal goals in mind. >> >> Juan's port was a "real PresentationManager/2 application", but it was a >> real PM/2 *Squeak* in that it opened a PM/2 window and BitBlt'd the >> Squeak Display to that. >> >> Opposite to that, the idea of "real PM/2" to me meant "Native Widgets" >> and I wanted to have that in Squeak.?? The problem was, in a live system >> (which is also "truly single-threaded"), how do I debug the >> "native-widget" event loop when the PM/2 is tied up by the BitBlt UI??? >> This meant that Juan's design wouldn't fit: I needed to somehow have all >> the Squeak tools but without touching PM/2.?? This was enabled by two >> things: (1) compiling the VM with Ian's sqXWindow.c, so Squeak would >> show on an X display while appearing to OS/2 initially as a headless app >> talking to the network, and (2) writing an FFI for the VM -- >> "native-widget" PM/2 calls were made via that FFI. >> >> >> >I wouldn't be surprised if it still works. But this was never part of >> >the official releases. >> I very much wanted to merge it upstream into Ian's unix port (the diff >> was very small).?? The desire seems weird and silly now, but back then I >> somehow thought it would make a difference, and I remember I was very >> frustrated that the merge didn't happen.?? But it *is* part of the 1.23 >> distribution on files.squeak.org today. >> >> >> >Later this was the base of his extremely interesting Cheese project. >> > >> Oh yes -- Cheese.?? This is what 3 years later led to the work on Eclipse >> SWT. >> >> >> >/I started with dissecting Slang out of the Cuis VMMaker packages and >> >VMMakerJS (SqeakJS) project into a couple of packages which could be >> >used for generating anything besides VM plugins./ >> You don't need Alien plugins to do VMIL18-simulation-style development >> of the core VM.?? For that there is now the Smalltalk implementation of >> RSP, which I recently added to VMMaker.?? There is a wide choice how to >> run the native CPU code: various simulators (gem5 or qemu), or on >> production silicon under GDB, or on some FPGA prototype under OpenOCD, >> ... well ... really anything that knows how to speak RSP.?? This will >> have a number of far-reaching consequences; this margin is too narrow to >> explain them, but you can watch a number of my presentations on the >> subject. >> >> Where does this stand right now? >> >> VMIL-18-simulation of Cog runs fine over RSP.?? VMMaker depends on a >> subtle inaccuracy in the Alien's emulation of instruction atomicity.?? >> This will cause VMMaker to fail on off-the-shelf production silicon.?? >> But when you modify the CPU to behave like Alien, OSTVM runs just fine >> (I verified on gem5).?? So it's a matter of very small fix to the Cogit >> to match what the stock processor is doing, before we have the full >> freedom to run anywhere we want. >> >> >> The larger problem that stops me from bringing in all the rest of the >> system (the target-agnostic synthesizer, the binutils-in-Smalltalk, the >> symbolic execution engine, the superoptimizer, etc), is the lack of a >> usable mother-Smalltalk. >> >> OSTVM is hosted in Squeak.?? With all my love for Squeak, I still can't >> use it for this kind of work.?? There are blocker problems with SUnit >> (causes erratic behavior of Socket code which works fine outside SUnit), >> so I can't have tests, I resort to launching everything from doits.?? I >> have no PetitParser.?? I have no JSON.?? I can't use my FFI-to-Z3 >> bindings.?? In the middle of working I randomly get unexpected problems >> with access to source.squeak.org.?? And so on. >> >> >> So. >> >> Is there an environment where work would not be blocked by such problems? >> >> Let's look. >> >> >Pharo VM did not only fork the transpiled C code, they also put the >> >Smalltalk VMMaker code in the repository and maintain it there. They >> >do not use the VMMaker Monticello repository if I understand >> >correctly. So in their repository and workflow, it really is a source >> >fork. >> You must know some magic!?? something I've been begging the Pharo people >> for months to tell me the secret where to download that source from.?? I >> tried the */headless/* branch, but it doesn't even compile in Pharo.?? I >> tried asking on pharo-dev, but I got zero responses since February. >> >> >> So. >> >> The million-dollar question again: >> >> Is there *any* environment that works well enough to do OST VM work in? >> >> >I started with dissecting Slang out of the Cuis VMMaker packages and >> >VMMakerJS (SqeakJS) project into a couple of packages which could be >> >used for generating anything besides VM plugins. >> > >> >Atleast that's my intentiont, I don't yet know of how that project >> >idea will pan out, atleast I intend to learn something of how Slang >> >works and how it's generated. >> > >> >It's currently highly expiremental, but I got some of the existing >> >VMMaker testscases "green" and so I thought I could share the current >> >state of it. Maybe the whole idea of using it outside the VMMaker is >> >not so a good idea, maybe, maybe not :-) >> > >> For the reasons I just explained above, I believe it would be more than >> just "a good idea".?? It is critical to the viability of this whole >> stack.?? If your project works, it give us an environment enabling >> further collaboration on OSTVM.?? Well, I must admit I don't know which >> way is shorter / more straightforward: this, or making Squeak work.?? I >> would be happy either way.?? I don't care which: I have much bigger fish >> to fry. >> >> -- boris >> >> -- Sent from: http://forum.world.st/Squeak-VM-f104410.html |
Update: Slang for Cuis (WIP, experimental)
|
|
In reply to this post by Boris Shingarov
On 5/19/20 1:05 PM, Boris Shingarov wrote: > > VMIL-18-simulation of Cog runs fine over RSP. VMMaker depends on a > subtle inaccuracy in the Alien's emulation of instruction atomicity. > This will cause VMMaker to fail on off-the-shelf production silicon. > But when you modify the CPU to behave like Alien, OSTVM runs just fine > (I verified on gem5). So it's a matter of very small fix to the Cogit > to match what the stock processor is doing, before we have the full > freedom to run anywhere we want. Update: This is now fixed. Thanks Eliot!!! |
Re: Slang for Cuis (WIP, experimental)
|
|
In reply to this post by Paul DeBruicker
Paul, David, Eliot, everyone: > And the latest versions (e.g. March 2013) of PetitParser & > PetitParserTests > on Lukas Rengglis site work too This is awesome news. When I last tried this (some time last year), it was failing, throwing me into a very depressed mood. Not that any of these issues are so bad or difficult in themselves, but when there is more work that needs to be done on the processor-modeling engine before it can really take off, than can fit in 24 hours/day, then looking at additional 100 small issues leads me to feel like, "ok forget it, there just isn't enough manpower". I am happy to learn that I am wrong about that. Ok. Let me give you a quick overview. I just committed a new VMMaker. Compared to the Sunday revision, this one runs not only in gem5, but also on the native desktop CPU without any CPU-emulation involved. So at least there is some point of integration that's working. It has, of course, obvious defects; for example, it is very slow. Part of it is because it's doing obviously-stupid things, e.g. only processing RSP S-packet when T-packet is available, indiscriminately asking for g- and G-transfers when neither is needed, doing unnecessary M-transfers, etc etc etc. A much larger reason is the frequency of semihosting requests (i.e. the segfaults on access to fake addresses; or what Modtalk calls "surgery points"). On the out-of-the-box SimpleAtomic gem5 model of x86, getting SpurReader to the first prompt takes 8935941 guest instructions in 15743835 clock cycles (7871917500ps simulated time) and 311777 segfaults: about 1 segfault per every 28 instructions. One interesting feature of "Target-Agnostic Modtalk" was the flexibility to configure, for each runtime subroutine, on which side of the segfault fence it will execute. In other words, if I have, say, all the lookup and send machinery production-ready, but I still need to write primitiveNew in simulation in the debugger, I can configure send/lookup code to execute natively -- i.e. with no segfaults involved -- but a jump to primitiveNew will segfault and get me into the Smalltalk debugger. But there is bigger fish to fry. This "RSP client" is a small part of "the ULD debugger" which in turn is part of "the target-agnostic verified engine". None of their operation involves the human programmer's expression of knowledge of the ISA: the system reals a formal specification of the processor, and the behavior of the tools is extracted from the proof. As it stands in revision 2747 today, the ULD inside Squeak is not connected to the formal model (which is just another Smalltalk object). To temporarily be able to make progress in combining with Cog/VMMaker, I use a mockup object representing a minimal set of facts about the x86 ISA just enough to allow execution over RSP -- but not any of the "really interesting" stuff such as symbolic execution (or even much simpler things like assembly/disassembly!) As I mentioned in my previous message, I got impeded by a few barriers to bringing the rest of the system over to Squeak; when we overcome those, the simplest next low-hanging fruit will be to repeat the same test runs on various ISAs without involving any mock-up objects, and exercising all the services VMMaker already asks from the Alien plugins (e.g., disassembly). Only slightly higher-hanging is notating the Cog Compiler in the retargetable assembler even before we bring in the actual superoptimizer. Even with manual instruction selection, the new assembler is able to propagate "holes" to the binary instruction stream. When this instruction-memory-with-holes is executed by the symbolic engine, the final state(s) is a term over the unknown variables, and in this way proof of program correctness in this case is possible even without involving superoptimization. But let's return to Earth. > Not sure about the Socket/SUnit > Here is my problem with SUnit: let's say I have some code acting as a client on a TCP socket. It connects to the TCP server, sends a few bytes, and receives a response. When I launch this as a DoIt, it always works. When I put it inside a TestCase, SUnit somehow interferes with the socket's receive semaphore, so the semaphore randomly (not always!) times out even when data is *already* available. The exactly same tests do not exhibit the problem on either Cuis or Pharo. Also, when on Squeak SUnit is not involved, that socket code never fails even for extremely complex exchanges on the network (such as running the whole Cog VM over RSP). But it would be very difficult to continue bringing over the other components without working SUnit (esp. considering how many tests are already there). |
«
Return to Squeak VM
|
1 view|%1 views
| Free forum by Nabble | Edit this page |