Spoon progress 15 April 2005: inert method deletion details and next steps

>
> Hi all--
>
> Now that I've been living with it for a while, here are a few details on
> the shrinking technique I mentioned a couple of months ago (where methods
> that haven't been run recently get reclaimed by the garbage collector).
> Also, at the end, are some notes on what I'm doing now and my next steps.
>
> When I did the earlier imprinting work, I added two bytes to the compiled
> method trailer format. One bit in those bytes indicates whether the
> virtual machine has run the method (the other fifteen are for recording
> the method's linear version). I changed the virtual machine to set that
> bit every time it runs a method. By clearing that bit on all methods, then
> examining them later after running the system for some duration, one can
> tell all the methods which were run over that duration. A method without
> that bit set is "inert".
>
> I extended the garbage collector with an alternative mark phase that
> doesn't mark or trace inert methods (with the exception of methods
> associated with currently-active contexts). References to inert methods
> are replaced with nil. This leaves some method dictionaries with nils
> where methods used to be.
>
> My typical development mode so far has been to use a "full" object memory
> equipped with remote-messaging tools to control a target object memory
> over a network. This lets me make changes to the target with impunity. In
> particular, I don't have to worry about the target getting wedged because
> I've broken the user interface, because I'm using another system's user
> interface. I've got a remote system browser, debugger, process browser,
> and workspace. The remote system browser uses the master system's
> compiler, and transfer methods directly into the target, so the target
> need not have a compiler (or ClassBuilder). No class names are ever
> exchanged between master and target, and source code is completely
> optional.
>
> I changed the target system's Object>>doesNotUnderstand: so that, before
> raising an exception, it first attempts to install the missing method from
> the connected master system. If installation is successful, it resends the
> method and carries on. I changed method lookup in the virtual machine so
> that when a nil is encountered where an inert method used to be, it is
> handled as a message-not-understood. So, the master system effectively
> acts as a virtual memory for the target, providing missing methods inline
> as they are encountered.
>
> In the past (before the garbage collector changes), a new target system
> was created as a copy of the master system. I would then connect master
> and target, and use the remote tools in the master to shrink the target
> manually. I've done two passes that way; the first took three months of
> work in 2003, the second took two weeks of work in 2005. The reason I've
> made multiple passes is that I kept realizing significant system features
> I'd forgotten to include before I started shrinking (e.g., remote
> debugging), and it'd be much more work to retrofit it into the target than
> to just shrink a new master copy. Since there will probably always be new
> fundamental things to add, I decided I ought to just make the shrinking
> process more automatic.
>
> Now I've got a class in the master that can automatically gut the target
> in 30 minutes. It invokes code in the target that can throw out entire
> "inert classes" (classes which have no references and no non-inert
> methods). I got rid of the entirety of Morphic this way, for example,
> without having to understand anything about how Morphic works.
>
> ***
>
> In January I wrote a module-aware webserver for the target, to which the
> web-based Spoon installer will redirect the user after downloading and
> starting the system. The user will then be able to discover and select
> modules to load and unload, make snapshots, quit, etc. Currently I'm
> working on a Naiad module which installs the ClassBuilder (for
> manipulating existing classes in the target). (Naiad is Spoon's module
> system.)
>
> After that I plan to:
>
> - make a module which installs every last bit of 3.8 final, just to show
> that one can recreate familiar old systems
>
> - make modules which install the VM construction tools
>
> - make a VM from a Spoon system with those modules in it
>
> - throw away the VMs and object memories that I started with!
>
> - make modules for other things I want to use (Quoth, Chronos, Weather on
> Display, Tweak, etc. etc.)
>
>
> Ideally, I'd like the new modules to contain well-factored and
> highly-readable expressions of the ideas of the old subsystems, rather
> than just blind repackagings, but we'll see... it's tempting to just
> imprint things to save time. :)  (For more about imprinting, see [1].)
>
>
> thanks,
>
> -C
>
> [1]
>
> http://lists.squeakfoundation.org/pipermail/spoon/2004-October/000061.html
>
> --
> Craig Latta
> improvisational musical informaticist
> www.netjam.org
> Smalltalkers do: [:it | All with: Class, (And love: it)]
>
>

>Hi all--
>
>         Now that I've been living with it for a while, here are a few
> details on the shrinking technique I mentioned a couple of months ago
> (where methods that haven't been run recently get reclaimed by the
> garbage collector). Also, at the end, are some notes on what I'm doing
> now and my next steps.
>
>         When I did the earlier imprinting work, I added two bytes to the
> compiled method trailer format. One bit in those bytes indicates whether
> the virtual machine has run the method (the other fifteen are for
> recording the method's linear version). I changed the virtual machine to
> set that bit every time it runs a method. By clearing that bit on all
> methods, then examining them later after running the system for some
> duration, one can tell all the methods which were run over that duration.
> A method without that bit set is "inert".
>
>         I extended the garbage collector with an alternative mark phase
> that doesn't mark or trace inert methods (with the exception of methods
> associated with currently-active contexts). References to inert methods
> are replaced with nil. This leaves some method dictionaries with nils
> where methods used to be.
>
>         My typical development mode so far has been to use a "full"
> object memory equipped with remote-messaging tools to control a target
> object memory over a network. This lets me make changes to the target
> with impunity. In particular, I don't have to worry about the target
> getting wedged because I've broken the user interface, because I'm using
> another system's user interface. I've got a remote system browser,
> debugger, process browser, and workspace. The remote system browser uses
> the master system's compiler, and transfer methods directly into the
> target, so the target need not have a compiler (or ClassBuilder). No
> class names are ever exchanged between master and target, and source code
> is completely optional.
>
>         I changed the target system's Object>>doesNotUnderstand: so that,
> before raising an exception, it first attempts to install the missing
> method from the connected master system. If installation is successful,
> it resends the method and carries on. I changed method lookup in the
> virtual machine so that when a nil is encountered where an inert method
> used to be, it is handled as a message-not-understood. So, the master
> system effectively acts as a virtual memory for the target, providing
> missing methods inline as they are encountered.
>
>         In the past (before the garbage collector changes), a new target
> system was created as a copy of the master system. I would then connect
> master and target, and use the remote tools in the master to shrink the
> target manually. I've done two passes that way; the first took three
> months of work in 2003, the second took two weeks of work in 2005. The
> reason I've made multiple passes is that I kept realizing significant
> system features I'd forgotten to include before I started shrinking
> (e.g., remote debugging), and it'd be much more work to retrofit it into
> the target than to just shrink a new master copy. Since there will
> probably always be new fundamental things to add, I decided I ought to
> just make the shrinking process more automatic.
>
>         Now I've got a class in the master that can automatically gut the
> target in 30 minutes. It invokes code in the target that can throw out
> entire "inert classes" (classes which have no references and no non-inert
> methods). I got rid of the entirety of Morphic this way, for example,
> without having to understand anything about how Morphic works.
>
>***
>
>         In January I wrote a module-aware webserver for the target, to
> which the web-based Spoon installer will redirect the user after
> downloading and starting the system. The user will then be able to
> discover and select modules to load and unload, make snapshots, quit,
> etc. Currently I'm working on a Naiad module which installs the
> ClassBuilder (for manipulating existing classes in the target). (Naiad is
> Spoon's module system.)
>
>         After that I plan to:
>
>-       make a module which installs every last bit of 3.8 final, just to
>show that one can recreate familiar old systems
>
>-       make modules which install the VM construction tools
>
>-       make a VM from a Spoon system with those modules in it
>
>-       throw away the VMs and object memories that I started with!
>
>-       make modules for other things I want to use (Quoth, Chronos,
>Weather on Display, Tweak, etc. etc.)
>
>
>         Ideally, I'd like the new modules to contain well-factored and
> highly-readable expressions of the ideas of the old subsystems, rather
> than just blind repackagings, but we'll see... it's tempting to just
> imprint things to save time. :)  (For more about imprinting, see [1].)
>
>
>         thanks,
>
>-C
>
>[1]
>
>http://lists.squeakfoundation.org/pipermail/spoon/2004-October/000061.html
>
>--
>Craig Latta
>improvisational musical informaticist
>www.netjam.org
>Smalltalkers do: [:it | All with: Class, (And love: it)]
>

> After that I plan to:
>
> - make a module which installs every last bit of 3.8 final, just to
> show
> that one can recreate familiar old systems
>
> - make modules which install the VM construction tools
>
> - make a VM from a Spoon system with those modules in it
>
> - throw away the VMs and object memories that I started with!
>
> - make modules for other things I want to use (Quoth, Chronos,
> Weather
> on Display, Tweak, etc. etc.)

> Which -I thought- the smiley at the end made clear. Sorry to see it  
> doesn't. Yes, of course, that was a dogmatic overstatement, it was  
> meant to be. But there is a grain of truth that's worthwhile to  
> discuss - namely that, if anything, tests should be used as "a"  
> primary source for imprinting (I'm putting the "a source" in quotes  
> to point out that I don't mean it to be the sole source just in  
> case someone else is inclined to interpret this as another dogmatic  
> overstatement which in that case it's not supposed to be ;-) <--  
> and please notice smiley here; this wasn't supposed to be taken too  
> seriously ;-) <-- etc.
>
> Cheers,
>   - Andreas
>