Hi folks,
What's the state of the art in shrinking these days? I'm starting a new web project with Squeak, and I want to create a smallish base image that I can use to build up images for development and deployment. It doesn't have to be absolutely minimal, just unloading unnecessary code and cleaning up the registries, clearing caches etc would fine.
I tried "Smalltalk shrinkAndCleanDesktop", but it complains about obsolete behaviours still being present, so I guess there's some bit rot there. Suggestions?
Colin |
On 6 January 2014 20:56, Colin Putney <[hidden email]> wrote:
> Hi folks, > > What's the state of the art in shrinking these days? > > I'm starting a new web project with Squeak, and I want to create a smallish > base image that I can use to build up images for development and deployment. > It doesn't have to be absolutely minimal, just unloading unnecessary code > and cleaning up the registries, clearing caches etc would fine. > > I tried "Smalltalk shrinkAndCleanDesktop", but it complains about obsolete > behaviours still being present, so I guess there's some bit rot there. > > Suggestions? Tobias Pape's the person to most recently attempt to shrink the image: I'd start by seeing what he stubbed his toes against. Tobias found that MC packages don't unload in the correct way: they do call a class's #unload (but only because MCClassDefinitions call their class's #removeFromSystem), but only do so after the class's methods have all been unloaded. That means that often the machinery needed to unload the class has already been unloaded. It looks... a bit messy to get things to work properly, possibly involving faking SystemChangeNotifier notifications of method removals. frank > Colin > > > |
In reply to this post by Colin Putney-3
On 06.01.2014, at 21:56, Colin Putney <[hidden email]> wrote: > Hi folks, > > What's the state of the art in shrinking these days? > > I'm starting a new web project with Squeak, and I want to create a smallish base image that I can use to build up images for development and deployment. It doesn't have to be absolutely minimal, just unloading unnecessary code and cleaning up the registries, clearing caches etc would fine. > > I tried "Smalltalk shrinkAndCleanDesktop", but it complains about obsolete behaviours still being present, so I guess there's some bit rot there. Before trying to shrink, check that there are _not_ the 8 Obsoletes I talked about in <[hidden email]> (http://forum.world.st/Obsoletes-tp4732904p4732931.html) Best -Tobias signature.asc (1K) Download Attachment |
On Mon, Jan 6, 2014 at 4:40 PM, Tobias Pape <[hidden email]> wrote: Before trying to shrink, check that there are _not_ the 8 Obsoletes I talked about in Aha! In the latest Trunk, there's only 6 obsolete classes, as the recent Environment changes fixed the two of them. The following script, adapted from your post above, cleans them up nicely:
" Fix obsolete ColorTheme" ColorTheme current class isObsolete ifTrue: [ ColorTheme current: nil. Smalltalk garbageCollect.
ColorTheme current]. " Fix obsolete uncle classes (cls->meta->super)" (Smalltalk allClasses select: [:cls | cls theMetaClass superclass isObsolete]
) copy do: [:withObsoleteUncle | | cls2 | cls2 := withObsoleteUncle addInstVarName: 'veryUnlikelyToExisitAnywhereInTheSystemAsVariableName'.
Smalltalk garbageCollect. cls2 removeInstVarName: 'veryUnlikelyToExisitAnywhereInTheSystemAsVariableName'.].
Smalltalk garbageCollect. Smalltalk fixObsoleteReferences. Running "Smalltalk shrink" then complains about two obsolete behaviours, and I this is a bug with Installer. If I just close the debugger and inspector, collect garbage and then run "SystemNavigation default obsoleteBehaviors" I get an empty array.
So it looks like the shrink works. It doesn't do much, though. My image goes from 20.9MB, down to 20.3MB. Colin |
On Mon, Jan 6, 2014 at 4:35 PM, Colin Putney <[hidden email]> wrote:
and (just to check) you've already done Smalltalk unloadReloadablePackages? I managed to get down to a 12 Mb minimal trunk but I had to hack fix a lot of MNUs. Basically we have work to do to make those packages really unload.
-- best,Eliot
|
In reply to this post by Colin Putney-3
Hi Colin-- I think the best way to shrink is for the garbage collector to do it. There's an alternate version of the GC in Spoon[1], invoked with Interpreter>>primitiveSnapshotWithoutStaleMethods. It sets a mark bit on all the methods associated with contexts in extant processes, along with any other methods which have run (for example, by running unit tests). Then it treats all other methods as garbage in the pre-snapshot GC. This leads to a chain reaction; the methods that aren't marked are usually the only things referring to their literals (including classes). The process is immediate, and requires no knowledge of what's being thrown away. The only difference between producing a minimal system (currently 1kB) and a larger one (e.g., the Spoon development base system at 300 kB) is how many methods you mark in advance by running them. -C [1] http://netjam.org/spoon/releases/current -- Craig Latta www.netjam.org/resume +1 510 984 8117 (Skype rings this until 31 January 2014) |
On 7 January 2014 02:33, Craig Latta <[hidden email]> wrote:
> > Hi Colin-- > > I think the best way to shrink is for the garbage collector to do > it. There's an alternate version of the GC in Spoon[1], invoked with > Interpreter>>primitiveSnapshotWithoutStaleMethods. It sets a mark bit on > all the methods associated with contexts in extant processes, along with > any other methods which have run (for example, by running unit tests). > Then it treats all other methods as garbage in the pre-snapshot GC. Craig, how difficult would it be to get the necessary VM changes for this feature into the main VMs? frank |
In reply to this post by Eliot Miranda-2
On Mon, Jan 6, 2014 at 9:15 PM, Eliot Miranda <[hidden email]> wrote:
Yup, #unloadReloadablePackages is part of #shrink. I was able to get down to about 19MB with no problems by unloading test packages. When I moved on to unloading production code that I don't need, I started getting MNUs too. I'll keep hacking at it until I see diminishing returns.
Thanks! Colin |
Don't forget there is no requirement to shrink an image along package
boundaries. Did you try, for example, removing TestCase and all subclasses? That would cut across package boundaries but so what? You can't unload Morphic but I'll bet you could dump a whole host of Morph subclasses. PaintBoxMorph, BouncingAtomsMorph, SoundRecorder, WaveEditor, PianoKeyboard, ScorePlayer, Storyboard, EventRecorder, Kedama and subclasses, etc., etc. etc. I think "running with a modified system" is one of Smalltalk's greatest and least appreciated strengths. On Tue, Jan 7, 2014 at 7:27 AM, Colin Putney <[hidden email]> wrote: > > > > On Mon, Jan 6, 2014 at 9:15 PM, Eliot Miranda <[hidden email]> > wrote: > >>> >>> So it looks like the shrink works. It doesn't do much, though. My image >>> goes from 20.9MB, down to 20.3MB. >> >> >> and (just to check) you've already done Smalltalk >> unloadReloadablePackages? I managed to get down to a 12 Mb minimal trunk >> but I had to hack fix a lot of MNUs. Basically we have work to do to make >> those packages really unload. > > > Yup, #unloadReloadablePackages is part of #shrink. I was able to get down to > about 19MB with no problems by unloading test packages. When I moved on to > unloading production code that I don't need, I started getting MNUs too. > I'll keep hacking at it until I see diminishing returns. > > Thanks! > > Colin > > > |
On Tue, Jan 7, 2014 at 11:40 AM, Chris Muller <[hidden email]> wrote: Don't forget there is no requirement to shrink an image along package I left TestCase in place, but I did try removing all subclasses. That led to an emergency evaluator. :-) It turns out I was mistaken. "Smalltalk unloadReloadablePackages" wasn't actually working, it bailed after the first few packages because it calls #fixObsoleteReferences after unloading a package, which raises an error if there are obsolete classes.
Now that I've got that sorted out, I can reliably shrink to a 11.3MB image using the attached script.
There's one oddity in that script. I had to patch SystemNavigation>>allObjectsDo:, because I was getting a #shouldNotImplement error from the SmallInteger implementation of #nextObject. Somehow, scanning the heap got to the end without encountering the sentinel object created at the top of #allObjectsDo:, and #nextObject answered 0. I'm not sure what to make of that.
Anyway, the script suggests a few fixes that should go into trunk before release. Colin shrink.st (5K) Download Attachment |
On 7 January 2014 17:32, Colin Putney <[hidden email]> wrote:
> > > > On Tue, Jan 7, 2014 at 11:40 AM, Chris Muller <[hidden email]> wrote: >> >> Don't forget there is no requirement to shrink an image along package >> boundaries. Did you try, for example, removing TestCase and all >> subclasses? That would cut across package boundaries but so what? > > > I left TestCase in place, but I did try removing all subclasses. That led to > an emergency evaluator. :-) > > It turns out I was mistaken. "Smalltalk unloadReloadablePackages" wasn't > actually working, it bailed after the first few packages because it calls > #fixObsoleteReferences after unloading a package, which raises an error if > there are obsolete classes. > > Now that I've got that sorted out, I can reliably shrink to a 11.3MB image > using the attached script. > > There's one oddity in that script. I had to patch > SystemNavigation>>allObjectsDo:, because I was getting a #shouldNotImplement > error from the SmallInteger implementation of #nextObject. Somehow, scanning > the heap got to the end without encountering the sentinel object created at > the top of #allObjectsDo:, and #nextObject answered 0. I'm not sure what to > make of that. > > Anyway, the script suggests a few fixes that should go into trunk before > release. I notice for starters that you #zapMVCProjects _after_ unloading ST80. Did you move that specifically, or was that just happenstance from folding #unloadReloadablePackages into the shrink script? Regarding this guy: names do: [:pkgName | (PackageInfo named: pkgName) classes do: [:ea | ea unload]]. names do: [:pkgName | Installer mc unload: pkgName]. That's pretty much what I was thinking of doing inside MC's #basicLoad. I hesitated because it involves sending #unload twice to every class, and doubted whether general implementations of #unload were idempotent. But I guess it must have worked out OK for you during the script? frank > Colin |
On Tue, Jan 7, 2014 at 2:28 PM, Frank Shearar <[hidden email]> wrote:
No, I think the zapping happens while ST80 is still there. I moved the list of packages to unload to the top, but left the actual unloading in place.
Regarding this guy: It seems to. But I thought the problem was that #unload has no effect, because the method has been removed by the time message is sent, and we fall back to the superclass noop. I originally had one loop the first sent #unload to all classes, than unloaded the package, but I found that was more sensitive to the order of the package names. The unload method would fail because it depended on something that's already gone. Two loops seems more robust.
I'm still working on it—down to about 10MB now. Will post again when I declare victory. Colin |
In reply to this post by Frank Shearar-3
Hi Frank-- > ...how difficult would it be to get the necessary VM changes for this > feature into the main VMs? It would be easy if the Unix VM build instructions were accurate. :) I wrote a separate message to vm-dev with details. The changes are at [1]. Please feel free to integrate them while I do, please let me know if you get somewhere. thanks, -C [1] http://tinyurl.com/p999omz -- Craig Latta www.netjam.org/resume +1 510 984 8117 (Skype rings this until 31 January 2014) |
In reply to this post by Colin Putney-3
> On Tue, Jan 7, 2014 at 11:40 AM, Chris Muller <[hidden email]> wrote:
>> >> Don't forget there is no requirement to shrink an image along package >> boundaries. Did you try, for example, removing TestCase and all >> subclasses? That would cut across package boundaries but so what? > > > I left TestCase in place, but I did try removing all subclasses. That led to > an emergency evaluator. :-) How did you try removing all subclasses? I wasn't sure how the system would respond to removing a class with subclasses, so I removed leaf subclasses first, which worked the first time. [(TestCase allSubclasses select: [ : each | each subclasses isEmpty ]) do: [ : each | each removeFromSystem: false ]. TestCase allSubclasses size > 0 ] whileTrue > It turns out I was mistaken. "Smalltalk unloadReloadablePackages" wasn't > actually working, it bailed after the first few packages because it calls > #fixObsoleteReferences after unloading a package, which raises an error if > there are obsolete classes. > > Now that I've got that sorted out, I can reliably shrink to a 11.3MB image > using the attached script. > > There's one oddity in that script. I had to patch > SystemNavigation>>allObjectsDo:, because I was getting a #shouldNotImplement > error from the SmallInteger implementation of #nextObject. Somehow, scanning > the heap got to the end without encountering the sentinel object created at > the top of #allObjectsDo:, and #nextObject answered 0. I'm not sure what to > make of that. > > Anyway, the script suggests a few fixes that should go into trunk before > release. Yes. Incidentally, sharing .st scripts through e-mail, Squeak can never seem to parse them when the attachment is downlaoded to another platform. I think because the email client treats it as "text" needing LE conversoins or something. Could you gzip your .st file and so it will be treated as binary, then I can look at it. OR, if you want to put it into trunk I'll just see it there. thanks! > > Colin > > > > > > > |
In reply to this post by Colin Putney-3
Many moons ago, John McIntosh was doing some shrinking for his initial work on getting Squeak to run on iOS.
This email refers to the work. There may have been more details to follow. Perhaps some of this might be helpful for your current shrinking work. Ken G. Brown
On Jan 7, 2014, at 12:38 PM, Colin Putney <[hidden email]> wrote:
|
In reply to this post by Colin Putney-3
John McIntosh did some shrinking work many moons ago when working on Squeak for iOS.
Perhaps this old email could help with your shrinking? Maybe there was further work after the email. Ken G. Brown
On Jan 7, 2014, at 12:38 PM, Colin Putney <[hidden email]> wrote:
|
On Tue, Jan 7, 2014 at 10:44 PM, Ken G. Brown <[hidden email]> wrote:
Nice! That shaved off another 700K, and brought the final shrink below 10MB.
Thanks. |
On Thu, Jan 9, 2014 at 6:02 PM, Colin Putney <[hidden email]> wrote:
|
On 9 January 2014 17:33, karl ramberg <[hidden email]> wrote:
> Se also Pavel Krivaneks changes; > http://forum.world.st/SqueakCore-td4661356.html I haven't had time to complete the job, but a lot of the recent work around shrinking, unloading and the like pretty much came either from me rediscovering Pavel's tricks, or me copying Pavel's tricks over to the image. frank > Cheers, > Karl > > > On Thu, Jan 9, 2014 at 6:02 PM, Colin Putney <[hidden email]> wrote: >> >> >> >> >> On Tue, Jan 7, 2014 at 10:44 PM, Ken G. Brown <[hidden email]> wrote: >>> >>> John McIntosh did some shrinking work many moons ago when working on >>> Squeak for iOS. >>> Perhaps this old email could help with your shrinking? Maybe there was >>> further work after the email. >> >> >> Nice! That shaved off another 700K, and brought the final shrink below >> 10MB. >> >> Thanks. >> >> >> > > > > |
On 1/9/14, Frank Shearar <[hidden email]> wrote:
> On 9 January 2014 17:33, karl ramberg <[hidden email]> wrote: >> Se also Pavel Krivaneks changes; >> http://forum.world.st/SqueakCore-td4661356.html > > I haven't had time to complete the job, but a lot of the recent work > around shrinking, unloading and the like pretty much came either from > me rediscovering Pavel's tricks, or me copying Pavel's tricks over to > the image. > > frank Excellent, thank you all and in particular, you, Frank. Hannes |
Free forum by Nabble | Edit this page |