[squeak-dev] ditching Monticello objects.

I'm building an image for a limited memory hardware platform.

At the point here where I want to *deploy* things and I can see that  
various parts of Monticello
are consuming well over a megabyte of memory to retain MCVersionInfo  
instances, MCMethodDefinition(s), time stamps, uuids, etc etc

I'm wondering if anyone has some code to strip all that out thus  
returning I suspect a couple of MB back for more constructive usage?

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================

John M McIntosh wrote:
> I'm building an image for a limited memory hardware platform.
>
> At the point here where I want to *deploy* things and I can see that
> various parts of Monticello
> are consuming well over a megabyte of memory to retain MCVersionInfo
> instances, MCMethodDefinition(s), time stamps, uuids, etc etc
>
> I'm wondering if anyone has some code to strip all that out thus
> returning I suspect a couple of MB back for more constructive usage?
Yep been there!

MC1.5 has a "trim ancestry" menu item in the right hand column of the
monticello browser. To get the ancestry back just reload the package, so
it is really redundant in a deployed image.

regards

Keith

I added a "trim ALL ancestry" which shows in the menu if no working
copies are selected.

Keith

John M McIntosh wrote:
> I'm building an image for a limited memory hardware platform.
>
> At the point here where I want to *deploy* things and I can see that
> various parts of Monticello
> are consuming well over a megabyte of memory to retain MCVersionInfo
> instances, MCMethodDefinition(s), time stamps, uuids, etc etc
>
> I'm wondering if anyone has some code to strip all that out thus
> returning I suspect a couple of MB back for more constructive usage?
You may want to look into the cleanUp/freeSomeSpace conventions. MC1.5
has a few implemented. There are many more available from
http://bugs.squeak.org/view.php?id=7244

Keith

Ok, well I did

I've been collecting lists of pruning methods so..

        ScrollBar initializeImagesCache.
        GradientFillStyle initPixelRampCache.
        NaturalLanguageFormTranslator classPool at: #CachedTranslations put:  
nil.
        NaturalLanguageTranslator resetCaches.
        PartsBin clearThumbnailCache.
        PaintBoxMorph classPool at: #ColorChart put: nil.
        PaintBoxMorph classPool at: #Prototype put: nil.
        ImageMorph classPool at: #DefaultForm put: (Form extent: 1@1 depth: 1).
        Utilities classPool at: #ScrapsBook put: nil.
        Project allInstancesDo: [ :each | each setThumbnail: nil ].
        ScriptingSystem stripGraphicsForExternalRelease.
        ProjectHistory currentHistory initialize.
        Utilities zapUpdateDownloader.
        Smalltalk forgetDoIts.
        Behavior flushObsoleteSubclasses.
        CommandHistory allInstancesDo: #initialize.
        ChangeSorter removeChangeSetsNamedSuchThat: [ :each | true ].
        ChangeSet resetCurrentToNewUnnamedChangeSet.
        MethodChangeRecord allInstancesDo: [ :x | x noteNewMethod: nil ].
        RequiredSelectors initialize.
        ProvidedSelectors initialize.
        LocalSends initialize.
        SendCaches initializeAllInstances.
        Utilities cleanseOtherworldlySteppers.
        Smalltalk organization removeEmptyCategories.
        Browser initialize.
        SMSqueakMap default purge.
        SystemBrowser removeObsolete.
        TheWorldMenu removeObsolete.
        AppRegistry removeObsolete.
        FileServices removeObsolete.
        MCFileBasedRepository flushAllCaches.
        MCDefinition clearInstances.
        Undeclared removeUnreferencedKeys.
        Smalltalk flushClassNameCache.
        3 timesRepeat: [ Smalltalk garbageCollect. Symbol compactSymbolTable ].
        Set rehashAllSets.
        Smalltalk garbageCollect.



I added these ones.

        ChangeSorter initializeChangeSetCategories.
        MCWorkingCopy inspect.
        MCRepositoryGroup inspect.
        MCCacheRepository inspect.

and cleaned up the class instance variables & any class vars via the  
inspect.

That removed all the MC packages and MC http repositories from the MC  
Browser

gave me back 4 megabytes of image space.

Well brute force, still 4 mb wonderful to get back this late in the  
project.


On 11-Feb-09, at 6:18 PM, Keith Hodges wrote:

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================

I've recently made a similar script for Pharo. See  
ScriptLoader>>cleanUpForRelease. This had brought the default image  
size below 10MB.

Adrian

On Feb 12, 2009, at 06:13 , John M McIntosh wrote:

John:
If you wish build a minimal image, I wish help.
See Esteban Lorenzano Mars ?
We working on it.
And I still can't get Squeak on my iPod Touch...

Edgar


On 2/12/09 3:13 AM, "John M McIntosh" <[hidden email]>
wrote:

Adrian Lienhard wrote:
> I've recently made a similar script for Pharo. See
> ScriptLoader>>cleanUpForRelease. This had brought the default image
> size below 10MB.
>
> Adrian

I give up, I really do, is this candid camera?

We have endless discussions on squeak-dev about how difficult it is to
clean up an image, we discuss,  suggest and implement conventions, to
solve the problem, that of implementing #freeSomeSpace on classes that
can non-destructively clean things up and #cleanUp on classes that can
destructively tidy themselves up before a release.

We had those discussions over 2 years ago.

What is the point?

Keith

Keith Hodges wrote: Sorry squeak folks, to explain, this is my frustration is with the Pharo
folks coming out, they continue to insist on being on a different planet.

This is how I think about things:

Hmm, perhaps we need a way of cleaning up the image before release.
Hmm, perhaps this would be useful to squeak users no matter what image
they are using.
Hmm,  perhaps people have done this work already somewhere,
Hmm, perhaps there is a body of knowledge we could contribute to as to
how to solve the problem for everyone in a consistent manner.

Perhaps, you never know,

I think I must be a an idealist trapped in a pragmatist body...

Keith

On Feb 12, 2009, at 15:15 , Keith Hodges wrote:

I think that's a good idea. I just don't expect someone else to do it.  
So don't expect me to do it. Maybe at some point I will get round to  
it, but at the moment I have higher priority items on my list.

Cheers,
Adrian

BTW, where is this implementation you mentioned? I cannot find it in  
3.10 and on the website I somehow can't find a download for a recent  
3.11 image.

But Adrian it has already been done, it was done years ago.

Keith

> Cheers,
> Adrian

Adrian Lienhard wrote: It was part of the Kernel-Extensions package, and now it is in.

http://bugs.squeak.org/view.php?id=7244

Keith

On Feb 12, 2009, at 15:52 , Keith Hodges wrote:

I was not aware of this Mantis task.

You say "it is in", but in where? I downloaded the latest image from  
the web page (3.10.2-7179-basic), but there are no implementors of  
#freeSomeSpace. Again, what do I miss? I guess, I somehow have to  
build this image, right? How are others supposed to reuse if there is  
not even a prebuilt image linked on the website ready to look into  
what you did?

Adrian

> Again, what do I miss? I guess, I somehow have to build this image,
> right? How are others supposed to reuse if there is not even a
> prebuilt image linked on the website ready to look into what you did?

Dear Adrian,

Ok, sorry I understand what you are saying.

with LevelPlayingField, and 3.11 we plan the tasks we want to have
included in the image in advance. Then fill in the details. Since
LevelPlayingField applies to any image you want it to, there is no image
for you to down load with it in. Nevertheless LPF has been around for a
while.

There is an image in the 3.11 ftp area called 3.10.2bc which is
essentially 3.10.2+LPF+Sake/Packages

For this fix all you have to do is Installer mantis ensureFix: 7244.

I will be updating it shortly, with John's great contributions

Keith

Great List John! I didn't realise that there were so many more. 96
Classes now respond to #cleanUp in my image.

http://bugs.squeak.org/view.php?id=7244

I added the following to the change set there.

Keith

====
>> I've been collecting lists of pruning methods so..
>>
>>     ScrollBar initializeImagesCache.
added
>>     GradientFillStyle initPixelRampCache.
added
>>     NaturalLanguageFormTranslator classPool at: #CachedTranslations
>> put: nil.
added
>>     NaturalLanguageTranslator resetCaches.
added
>>     PartsBin clearThumbnailCache.
already present
>>     PaintBoxMorph classPool    at: #ColorChart put: nil.
added
>>     PaintBoxMorph classPool at: #Prototype put: nil.
added
>>     ImageMorph classPool at: #DefaultForm put: (Form extent: 1@1
>> depth: 1).
added
>>     Utilities classPool at: #ScrapsBook put: nil.
added #cleanUp
>>     Project allInstancesDo: [ :each | each setThumbnail: nil ].
not sure about this one
>>     ScriptingSystem stripGraphicsForExternalRelease.
added to StadardScriptingSystem cleanUp
>>     ProjectHistory currentHistory initialize.
added
>>     Utilities zapUpdateDownloader.
added
>>     Smalltalk forgetDoIts.
>>     Behavior flushObsoleteSubclasses.
>>     CommandHistory allInstancesDo: #initialize.
added
>>     ChangeSorter removeChangeSetsNamedSuchThat: [ :each | true ].
equivalent already in ChangeSet #cleanUp
>>     ChangeSet resetCurrentToNewUnnamedChangeSet.
updated ChangeSet #cleanUp
>>     MethodChangeRecord allInstancesDo: [ :x | x noteNewMethod: nil ].
>>     RequiredSelectors initialize.
currently does:  self current ensureClean
>>     ProvidedSelectors initialize.
added ModelExtension cleanUp
>>     LocalSends initialize.
included in ModelExtension cleanUp
>>     SendCaches initializeAllInstances.
added cleanUp
>>     Utilities cleanseOtherworldlySteppers.
already in Utilities freeSomeSpace
>>     Smalltalk organization removeEmptyCategories.
added to ClassOrganizer cleanUp
>>     Browser initialize.
>>     SMSqueakMap default purge.
already in SMSqueakMap freeSomeSpace
>>     SystemBrowser removeObsolete.
covered by AppRegistry freeSomeSpace
>>     TheWorldMenu removeObsolete.
>>     AppRegistry removeObsolete.
covered by AppRegistry removeObsolete
>>     FileServices removeObsolete.
>>     MCFileBasedRepository flushAllCaches.
>>     MCDefinition clearInstances.
>>     Undeclared removeUnreferencedKeys.
In SmalltalkImage cleanUp
>>     Smalltalk flushClassNameCache.
in SystemDictionary cleanup
>>     3 timesRepeat: [ Smalltalk garbageCollect. Symbol
>> compactSymbolTable ].
>>     Set rehashAllSets.
does this save memory?

On 2/12/09 12:15 PM, "Keith Hodges" <[hidden email]> wrote:

> What is the point?
>
> Keith

To prove they also could shrink a image, not matter how many of us do same
before

Edgar

2009/2/12 Keith Hodges <[hidden email]>: you're not alone :)

> Keith
>
>
>



--
Best regards,
Igor Stasenko AKA sig.

Keith Hodges wrote:
>
> For this fix all you have to do is Installer mantis ensureFix: 7244.
>
> I will be updating it shortly, with John's great contributions
>
> Keith
>  
OK, you piqued my interest.  FYI If I start with a
Squeak3.9.1-final-7075 image, install LPF, apply this "fix" and then do:

SmalltalkImage current cleanUpAll

I get the attached walkback.  I haven't investigated it any further.

David



MessageNotUnderstood: UndefinedObject>>initializeChangeSetCategories
12 February 2009 2:51:29 pm

VM: unix - a SmalltalkImage
Image: Squeak3.9.1 [latest update: #7075]

SecurityManager state:
Restricted: false
FileAccess: true
SocketAccess: true
Working Dir /home/shaffer/Source/Squeak/ProfitPortal/builds/6
Trusted Dir /home/shaffer/Source/Squeak/ProfitPortal/builds/6/secure
Untrusted Dir /home/shaffer/Source/Squeak/ProfitPortal/builds/6/My Squeak

UndefinedObject(Object)>>doesNotUnderstand: #initializeChangeSetCategories
        Receiver: nil
        Arguments and temporary variables:
                aMessage: initializeChangeSetCategories
        Receiver's instance variables:
nil

UndefinedObject>>cleanUp
        Receiver: nil
        Arguments and temporary variables:

        Receiver's instance variables:
nil

[] in SmalltalkImage>>cleanUpAllExcept: {[:ea | ea cleanUp]}
        Arguments and temporary variables:
                aListOfClassNames: #()
                exclude: a Set()
                ea: nil

[] in SystemDictionary(Dictionary)>>do: {[:assoc | aBlock value: assoc value]}
        Arguments and temporary variables:
                aBlock: [] in SmalltalkImage>>cleanUpAllExcept: {[:ea | ea cleanUp]}
                assoc: #ActiveEvent->nil


--- The full stack ---
UndefinedObject(Object)>>doesNotUnderstand: #initializeChangeSetCategories
UndefinedObject>>cleanUp
[] in SmalltalkImage>>cleanUpAllExcept: {[:ea | ea cleanUp]}
[] in SystemDictionary(Dictionary)>>do: {[:assoc | aBlock value: assoc value]}
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
SystemDictionary(Set)>>do:
SystemDictionary(Dictionary)>>do:
SystemDictionary(Collection)>>select:thenDo:
SmalltalkImage>>cleanUpAllExcept:
SmalltalkImage>>cleanUpAll
UndefinedObject>>DoIt
Compiler>>evaluate:in:to:notifying:ifFail:logged:
Compiler class>>evaluate:for:notifying:logged:
Compiler class>>evaluate:for:logged:
Compiler class>>evaluate:logged:
[] in RWBinaryOrTextStream(PositionableStream)>>fileInAnnouncing: {[val := (self peekFor: $!)     ifTrue: [(Compiler evaluate: self nextChunk l...]}
BlockContext>>on:do:
[] in RWBinaryOrTextStream(PositionableStream)>>fileInAnnouncing: {[:bar |  [self atEnd]   whileFalse: [bar value: self position.    self skipS...]}
[] in ProgressInitiationException>>defaultMorphicAction {[result := workBlock value: progress]}
BlockContext>>ensure:
ProgressInitiationException>>defaultMorphicAction
ProgressInitiationException>>defaultAction
UndefinedObject>>handleSignal:
ProgressInitiationException(Exception)>>signal
ProgressInitiationException>>display:at:from:to:during:
ProgressInitiationException class>>display:at:from:to:during:
ByteString(String)>>displayProgressAt:from:to:during:
RWBinaryOrTextStream(PositionableStream)>>fileInAnnouncing:
RWBinaryOrTextStream(PositionableStream)>>fileIn
CodeLoader>>installSourceFile:
[] in CodeLoader>>installSourceFiles {[:req | self installSourceFile: req contentStream]}
Array(SequenceableCollection)>>do:
CodeLoader>>installSourceFiles
ProjectLauncher>>startUpAfterLogin
ProjectLauncher>>startUp
[] in AutoStart class>>startUp: {[launcher startUp]}
WorldState>>runStepMethodsIn:
PasteUpMorph>>runStepMethods
WorldState>>doOneCycleNowFor:
WorldState>>doOneCycleFor:
PasteUpMorph>>doOneCycle
[] in Project class>>spawnNewProcess {[[World doOneCycle.  Processor yield.  false] whileFalse.  nil]}
[] in BlockContext>>newProcess {[self value.  Processor terminateActive]}

On 12-Feb-09, at 2:22 AM, Edgar wrote:

> John:
> If you wish build a minimal image, I wish help.
> See Esteban Lorenzano Mars ?
> We working on it.

Ok, well I can send you an image at some point so you can see what I  
removed.


> And I still can't get Squeak on my iPod Touch...

So what problem are you having?


> Edgar

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================

Ok, there still are a few more subclasses in the MC hierarchy that  
need cleanup, one that deals with cached file based repositories for  
example.
However when dealing with the cleanup code likely it could be class  
allsubclasses do:   kind of method.

Also PackageOrganizer  seems to hold onto something.



On 12-Feb-09, at 9:46 AM, Keith Hodges wrote:

> Great List John! I didn't realise that there were so many more. 96
> Classes now respond to #cleanUp in my image.
>
> http://bugs.squeak.org/view.php?id=7244
>
> I added the following to the change set there.
>
> Keith

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================